How to upgrade the SSD in your Palo Alto PA-200 to extend it's life.

I love my PA-200s. They’re great for learning Palo Alto and they’re available cheap on ebay, but one common problem I have had with them is that the storage space is just too small. There has never been enough room to store more than a couple images, and recently the root partition filled up on one of my devices, rendering the device unbootable. 

I’ve performed this upgrade before, but I thought it would be nice to document it in case anyone else wanted to do the upgrade and extend the life of these handy little firewalls. This will give you more room for images, captures, etc. It will not make the device boot faster though, except for the instances where disk cleanup is required during boot to get it to boot at all, sorry.

The sign that your firewall is now a brick.

2019-10-29T23:37:07-05:00 PA-200 1,2019/10/29 23:37:07,xxxxxxxxxxxxxx,SYSTEM,general,0,2019/10/29 23:37:07,,general,,0,0,general,critical,"Disk usage for / exceeds limit, 100 percent in use, cleaning filesystem",235026,0x0,0,0,0,0,,PA-200

2019-10-29T23:40:07-05:00 PA-200 1,2019/10/29 23:40:07,xxxxxxxxxxxxxx,SYSTEM,general,0,2019/10/29 23:40:07,,general,,0,0,general,critical,"Disk usage for / exceeds limit, 100 percent in use, cleaning filesystem",235027,0x0,0,0,0,0,,PA-200

2019-10-29T23:43:07-05:00 PA-200 1,2019/10/29 23:43:07,xxxxxxxxxxxxxx,SYSTEM,general,0,2019/10/29 23:43:07,,general,,0,0,general,critical,"Disk usage for / exceeds limit, 100 percent in use, cleaning filesystem",235028,0x0,0,0,0,0,,PA-200

2019-10-29T23:46:07-05:00 PA-200 1,2019/10/29 23:46:07,xxxxxxxxxxxxxx,SYSTEM,general,0,2019/10/29 23:46:07,,general,,0,0,general,critical,"Disk usage for / exceeds limit, 100 percent in use, cleaning filesystem",235029,0x0,0,0,0,0,,PA-200

2019-10-29T23:51:00-05:00 PA-200 1,2019/10/29 23:51:00,xxxxxxxxxxxxxx,SYSTEM,general,0,2019/10/29 23:51:00,,general,,0,0,general,critical,"Disk usage for / exceeds limit, 100 percent in use, cleaning filesystem",235030,0x0,0,0,0,0,,PA-200

2019-10-29T23:54:00-05:00 PA-200 1,2019/10/29 23:54:00,xxxxxxxxxxxxxx,SYSTEM,general,0,2019/10/29 23:54:00,,general,,0,0,general,critical,"Disk usage for / exceeds limit, 100 percent in use, cleaning filesystem",235031,0x0,0,0,0,0,,PA-200

2019-10-29T23:57:00-05:00 PA-200 1,2019/10/29 23:57:00,xxxxxxxxxxxxxx,SYSTEM,general,0,2019/10/29 23:57:00,,general,,0,0,general,critical,"Disk usage for / exceeds limit, 100 percent in use, cleaning filesystem",235032,0x0,0,0,0,0,,PA-200

As you can see, the firewall attempted to clean up the filesystem, but was never successful in cleaning it up enough to successfully boot up. In this instance, I let it ‘try’ for 12 hours since it was already late and I didn’t feel like performing surgery at that time, and I was just curious if it would actually be able to get through it. It didn’t, so the next day, I upgraded the SSD. It’s possible to just clean up the root partition and get some space back, but if you are going through the trouble of pulling the SSD out, you might as well take the opportunity to fix it permanently.

You may see these errors while the firewall is running as well. In my case, the / partition being full was preventing me from logging in, which I didn’t realize until I rebooted it to resolve the login issue. 

Disassembling the device is straight forward. There are only 4 screws that need to be removed. One on each side of the front bezel and 2 on the bottom of the bezel. After those screws have been removed, slide the bezel forward and lift it off. If it doesn’t slide freely, it’s because there is a piece of double sided tape on the underside of the cover that sticks to the heat sink. You’ll have to pry it apart in this case, by prying the front bezel forward and up until you can free it from the heat sink. The back of the top cover cannot be pried up because there’s a lip holding it down to the lower half of the case. You can see the bezel was bent a little in this photo, but it is easy to bend back in shape if you don’t go crazy pulling it off.

Now that you have it apart, you’ll see the SSD that’s causing you grief. The standard SSD is just 16G and is partitioned up so that you just don’t have much room to work with. I’m going to replace this one with a 120G SSD, which should last the rest of this little guy’s life.

You will need to remove the mounting brackets and then remove the SSD from the tray.

Remember to reconnect the ground wire when you put the new drive in.

For this writeup, you’ll need a computer that you can mount both the drives in. I am going to use a CentOS server because that’s what I have ready on my bench, but anything that can mount EXT3 should work.

# parted /dev/sdb

GNU Parted 3.1

Using /dev/sdb

Welcome to GNU Parted! Type 'help' to view a list of commands.

(parted) p                                                               

Model: ATA Virtium - TuffDi (scsi)

Disk /dev/sdb: 15.9GB

Sector size (logical/physical): 512B/512B

Partition Table: msdos

Disk Flags:

Number  Start   End     Size    Type      File system  Flags

 1      32.3kB  8225kB  8193kB  primary   ext3

 2      8225kB  2065MB  2056MB  primary   ext3

 3      2065MB  4121MB  2056MB  primary   ext3

 4      4121MB  15.9GB  11.8GB  extended

 5      4121MB  11.3GB  7172MB  logical   ext3

 6      11.3GB  13.3GB  2056MB  logical   ext3

 7      13.3GB  13.4GB  8193kB  logical

 8      13.4GB  15.9GB  2574MB  logical   ext3

The partitions are laid out like this.

1 = maintenance

This has been empty on every one that I’ve looked at.

2 = sysroot0

This is the primary OS partition (version 7.1.3 in this case)

3 = sysroot1

This is the fallback OS partition (version 7.1.0 in this case)

5 = Config

6 = Repository

The images you have stored on the device are located here.

7 = swap

8 = Logs

Device logs are here.

For the new device, I am going to create partition 1 at 16mb

Partition 3 and 4 will be 8G

Partition 4 is extended and just covers the rest of the disk (95.8G).

Partition 5 and 6 will be 16G, which will be more than enough for the config and plenty of extra room for a larger repository to make it easier to perform upgrades if you are lucky enough to still have a contract.

Partition 7 will stay 8MB, because it’s just swap. If you need more memory, you could increase this I suppose, but I haven’t had memory issues in any of my PA-200s yet. Remember to set the type to swap when creating this partition.

Partition 8 gets whatever is left. Although I have never run out of room here, it seems like the logical place to spend your extra disk space and should allow you to increase your logging levels to maximum everywhere. I send all of my logs to syslog, but the built in features of the PA are very handy too.

I’ll use fdisk for this.

# fdisk /dev/sdc

Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only until you write them.

Be careful before using the write command.

Command (m for help): o

Building a new DOS disklabel with disk identifier 0xc01cef80.

Command (m for help): p

Disk /dev/sdc: 120.0 GB, 120040980480 bytes, 234455040 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk label type: dos

Disk identifier: 0xc01cef80

   Device Boot      Start         End      Blocks   Id  System

Command (m for help): n

Partition type:

   p   primary (0 primary, 0 extended, 4 free)

   e   extended

Select (default p): p

Partition number (1-4, default 1):

First sector (2048-234455039, default 2048):

Using default value 2048

Last sector, +sectors or +size{K,M,G} (2048-234455039, default 234455039): +16M

Partition 1 of type Linux and of size 16 MiB is set

Command (m for help): n

Partition type:

   p   primary (1 primary, 0 extended, 3 free)

   e   extended

Select (default p): p

Partition number (2-4, default 2): 2

First sector (34816-234455039, default 34816):

Using default value 34816

Last sector, +sectors or +size{K,M,G} (34816-234455039, default 234455039): +8G

Partition 2 of type Linux and of size 8 GiB is set

Command (m for help): n

Partition type:

   p   primary (2 primary, 0 extended, 2 free)

   e   extended

Select (default p): p

Partition number (3,4, default 3): 3

First sector (16812032-234455039, default 16812032):

Using default value 16812032

Last sector, +sectors or +size{K,M,G} (16812032-234455039, default 234455039): +8G

Partition 3 of type Linux and of size 8 GiB is set

Command (m for help): n

Partition type:

   p   primary (3 primary, 0 extended, 1 free)

   e   extended

Select (default e): e

Selected partition 4

First sector (33589248-234455039, default 33589248):

Using default value 33589248

Last sector, +sectors or +size{K,M,G} (33589248-234455039, default 234455039):

Using default value 234455039

Partition 4 of type Extended and of size 95.8 GiB is set

Command (m for help): n

All primary partitions are in use

Adding logical partition 5

First sector (33591296-234455039, default 33591296):

Using default value 33591296

Last sector, +sectors or +size{K,M,G} (33591296-234455039, default 234455039): +16G

Partition 5 of type Linux and of size 16 GiB is set

Command (m for help): n

All primary partitions are in use

Adding logical partition 6

First sector (67147776-234455039, default 67147776):

Using default value 67147776

Last sector, +sectors or +size{K,M,G} (67147776-234455039, default 234455039): +16G

Partition 6 of type Linux and of size 16 GiB is set

Command (m for help): n

All primary partitions are in use

Adding logical partition 7

First sector (100704256-234455039, default 100704256):

Using default value 100704256

Last sector, +sectors or +size{K,M,G} (100704256-234455039, default 234455039): +8M

Partition 7 of type Linux and of size 8 MiB is set

Command (m for help): t

Partition number (1-7, default 7): 7

Hex code (type L to list all codes): 82

Changed type of partition 'Linux' to 'Linux swap / Solaris'

Command (m for help): n

All primary partitions are in use

Adding logical partition 8

First sector (100722688-234455039, default 100722688):

Using default value 100722688

Last sector, +sectors or +size{K,M,G} (100722688-234455039, default 234455039):

Using default value 234455039

Partition 8 of type Linux and of size 63.8 GiB is set

Command (m for help): p

Disk /dev/sdc: 120.0 GB, 120040980480 bytes, 234455040 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk label type: dos

Disk identifier: 0xc01cef80

   Device Boot      Start         End      Blocks   Id  System

/dev/sdc1            2048       34815       16384   83  Linux

/dev/sdc2           34816    16812031     8388608   83  Linux

/dev/sdc3        16812032    33589247     8388608   83  Linux

/dev/sdc4        33589248   234455039   100432896    5  Extended

/dev/sdc5        33591296    67145727    16777216   83  Linux

/dev/sdc6        67147776   100702207    16777216   83  Linux

/dev/sdc7       100704256   100720639        8192   82  Linux swap / Solaris

/dev/sdc8       100722688   234455039    66866176   83  Linux

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

# lsblk /dev/sdc

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT

sdc      8:32   0 111.8G  0 disk

   sdc1   8:33   0    16M  0 part

   sdc2   8:34   0     8G  0 part

   sdc3   8:35   0     8G  0 part

   sdc4   8:36   0     1K  0 part

   sdc5   8:37   0    16G  0 part

   sdc6   8:38   0    16G  0 part

   sdc7   8:39   0     8M  0 part

   sdc8   8:40   0  63.8G  0 part

Now you must initialize each of the EXT partitions.

  mkfs.ext3 /dev/sdc1

  mkfs.ext3 /dev/sdc2

  mkfs.ext3 /dev/sdc3

  mkfs.ext3 /dev/sdc5

  mkfs.ext3 /dev/sdc6

  mkfs.ext3 /dev/sdc8

Now you must initialize the swap partition.

  mkswap /dev/sdc7

Now it is time to mount everything and copy old to new.

I’m just going to create a directory under /mnt for each partition and mount them there for simplicity

Filesystem               Size  Used Avail Use% Mounted on

/dev/sdb1                6.6M   51K  6.2M   1% /mnt/sdb1

/dev/sdb2                1.9G  1.8G     0 100% /mnt/sdb2

/dev/sdb3                1.9G  1.2G  578M  68% /mnt/sdb3

/dev/sdb5                6.5G  1.7G  4.5G  28% /mnt/sdb5

/dev/sdb6                1.9G  656M  1.2G  37% /mnt/sdb6

/dev/sdb8                2.3G  1.2G  1.1G  54% /mnt/sdb8

/dev/sdc1                 15M  145K   14M   2% /mnt/sdc1

/dev/sdc2                7.8G   19M  7.4G   1% /mnt/sdc2

/dev/sdc3                7.8G   19M  7.4G   1% /mnt/sdc3

/dev/sdc5                 16G   45M   15G   1% /mnt/sdc5

/dev/sdc6                 16G   45M   15G   1% /mnt/sdc6

/dev/sdc8                 63G   52M   60G   1% /mnt/sdc8

Use “cp -pr” to preserve permissions and timestamps and to act recursively

 #cp -pr sdb1/* sdc1

 #cp -pr sdb2/* sdc2

 #cp -pr sdb3/* sdc3

 #cp -pr sdb5/* sdc5

 #cp -pr sdb6/* sdc6

 #cp -pr sdb8/* sdc8

Filesystem               Size  Used Avail Use% Mounted on

/dev/sdc1                 15M  145K   16M   2% /mnt/sdc1

/dev/sdc2                7.8G  1.9G  5.6G  25% /mnt/sdc2

/dev/sdc3                7.8G  1.3G  6.2G  17% /mnt/sdc3

/dev/sdc5                 16G  1.8G   14G  12% /mnt/sdc5

/dev/sdc6                 16G  697M   15G   5% /mnt/sdc6

/dev/sdc8                 63G  1.3G   59G   3% /mnt/sdc8

Now you are almost done.

A quick examination of fstab from the PA-200 shows that mounts are done via labels, so we need to label the new partitions.

# cat fstab

# PAN version 7.1.3

LABEL=sysroot0  /                       ext3    defaults    1 1

LABEL=pancfg    /opt/pancfg             ext3    defaults    1 2

LABEL=panrepo   /opt/panrepo            ext3    defaults    1 2

/dev/sda7       swap                    swap    defaults    0 0

proc            /proc                   proc    defaults    0 0

tmpfs           /dev/shm                tmpfs   defaults    0 0

devpts          /dev/pts                devpts  defaults    0 0

sys             /sys                    sysfs   defaults    0 0

nfsd            /proc/fs/nfsd           nfsd    defaults,auto 0 0

sunrpc          /var/lib/nfs/rpc_pipefs rpc_pipefs defaults,auto 0 0

While fstab only appears to be using a couple of labels, the other partitions are also labeled so it would be safest to go ahead and relabel all of the new partitions as well.

[CentOS etc]# e2label /dev/sdb1

maint

[CentOS etc]# e2label /dev/sdb2

sysroot0

[CentOS etc]# e2label /dev/sdb3

sysroot1

 [CentOS etc]# e2label /dev/sdb5

pancfg

[CentOS etc]# e2label /dev/sdb6

panrepo

 [CentOS etc]# e2label /dev/sdb8

Panlogs

[CentOS etc]# e2label /dev/sdc1 maint

[CentOS etc]# e2label /dev/sdc2 sysroot0

[CentOS etc]# e2label /dev/sdc3 sysroot1

[CentOS etc]# e2label /dev/sdc5 pancfg

[CentOS etc]# e2label /dev/sdc6 panrepo

[CentOS etc]# e2label /dev/sdc8 Panlogs

Umount everything, power down and put the new SSD in your PA-200

Remember to re-connect the ground wire!

Button up the case, reconnect everything, power up and wait the customary “forever” for it to boot up, and rejoice at the free space you now have.

This device took just under 20 minutes to boot and complete the autocommit job after this upgrade, which is about normal for a good reboot of a PA-200.

Log allocation is handled by percentages of disk space by default. If you have modified this you may want to reset your changes now that you have more space available.

Note. If you move a SSD from one PA-200 to another PA-200, you will need to update a udev rule in sysroot0 and sysroot1. Comment the following line out of /etc/udev/rules.d/70-persistent-net.rules. The next boot will recreate it correctly, otherwise the new MAC will be added for eth1 and it will not boot.

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="b4:0c:25:xx:xx:xx", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"