AIX Service Strategy Details and Best Practices
Introduction
IBM has enhanced the AIX operating system (OS) Release and Service Strategy as part of an ongoing effort to improve the manageability and stability of the AIX operating system for our clients. The enhanced strategy will provide clients with:
• Longer support for each AIX Technology Level update (formerly known as Recommended Maintenance Levels)
• Improved serviceability for AIX throughout the life of each Technology Level
• Support for some new hardware on existing Technology Levels
The previous paragraph came from the IBM AIX Operating System Release Service Strategy paper, which should be read before reading this paper.
This paper will go deeper into some of the changes, give some best practice scenarios and discuss some of the more advanced function available.
Enhanced Service Strategy Releases
The AIX enhanced service strategy will start with AIX 5.3 TL6 and continue with AIX 6.1. AIX 5.2 will not use the new service strategy; TL10 is the last technology level available on 5.2 and we will continue to provide service packs until the end of support.
Contents of a Service Pack
Service packs contain fixes for:
• Customer reported problems (APARs) that cannot wait until the next TL
• Critical problems found by development or test teams
• Very, very limited number of changes to support new hardware. Examples: A new device driver, a new ODM entry to allow for configuration of a new class or type of device, small changes in the kernel to recognize a new processor speed, etc. The only changes that are allowed in an SP are limited to minimal corrections that do not change behavior or add new functionality. The development team employs a fix rating system to
enforce this. New function, both for hardware exploitation and software features, is only shipped in Technology Levels or new releases.
While AIX TL’s contain RSCT and CSM fileset updates, those filesets are not updated in the AIX Service Packs. The reason for this is scheduling. The RSCT and CSM teams line up their schedules to meet the AIX TL schedules, but do not do the same for Service Packs. The fixes are placed into the next TL. Occasionally, usually with applications or products that rely more heavily on CSM or RSCT, you may be required to download a fix directly from the RSCT or CSM download sites. This situation may be more suited for your environment because it does not require application of a specific AIX SP, only the individual fix for CSM or RSCT. We are continuing to explore possibilities here to simplify the maintenance of AIX and its components.
AIX Service Strategy Details and Best Practices
Changing the names of the service packs
Starting with TL6, Service Packs will be labeled with their release date, using the YYWW
format, where:
YY = 2 digit year
WW = 2 digit week
An example: AIX 5.3 TL6 SP3 was released in 2007 and the 32nd week of the year, therefore:
5300-06-03-0732
‘ ‘ ‘ ‘———————YYWW
‘ ‘ ‘—————–SP3
‘ ‘————TL level
‘–AIX release
The ‘oslevel –s’ command was updated to report this new name. The nice thing about this is not only can you tell when the SP was released, but you will also know which SP to move to on a new TL. When moving up to a new TL, you must move to a SP that is the same or later than your current SP. The SP number itself (IE SP3) will not be the same, because the Service Packs will be numbered consecutively as they are released, but the dates will tell you where you need to be on the new TL. In general, it is recommended that you apply the latest Service Pack when moving to a new TL.
Service pack names will be changed to reflect the release dates:
VR00-TL-SP-YYWW 5300-07-04-0812
We will line SP’s up so the upgrade path will be clear
AIX Service Strategy Details and Best Practices
Applying individual PTF updates after a Service Pack will not be shown with the ‘oslevel –s’ command. Instead, use ‘oslevel –s –g 5300-06-03-0732’ to show what filesets are greater than the specified service pack. The installp command was changed, as were the updates themselves, to not allow a system to apply any updates that are ‘younger’ than what is currently installed. This would cause regression. When moving to a new TL, we recommend moving to the latest SP, that way you are guaranteed it will pass the installp regression test.
Using the example above, if you are currently at TL7 SP8 (5300-07-08-0845), then you would not be allowed to install TL8 SP2 (5300-08-02-0830), because 0845 was released after 0830, and therefore will have fixes and hardware enablement that 0830 does not have.
The oslevel –s command will now print out the new SP format, but the other options will not change.
# oslevel
5300
# oslevel –r
5300-06
# oslevel –s
5300-06-03-0736
The installp command will also stop an update entirely if it sees any updates that are older (released before) in the list it is trying to apply. This is to make sure that part of a SP is not installed, if that was not your intention. If you see a message from installp about not being able
to install a fileset update because of regression, then go to a newer SP (the latest) and try the install again.
Changing the M in VRMF
Starting with AIX 5.3 TL7, any filesets that are updated will get a new ‘M’ in the VRMF. For example, if the bos.rte.lvm fileset was updated, its update in TL7 would be 5.3.7.0 and the first update, in SP1, will be 5.3.7.1. The fileset updates won’t necessarily correspond to the SP
because an update for fileset bos.rte.install might not come out until SP3, but it would be called bos.rte.install 5.3.7.1. Likewise, if bos.rte.lvm had not changed since SP1, it would still be bos.rte.lvm 5.3.7.1 in SP3.
Technology Levels Must be Applied as a Group
Technology Levels must be applied as a group, using the ‘smitty update_all’ or ‘install_all_updates’ commands. Installing a Technology Level is an “all or nothing” operation.
Initially, the plan was to add requisites to glue the TL together, but this was not done because of the complications of circular requisites. But, installing a partial Technology Level will not be recognized from a support standpoint. Before applying a TL, you should always create a backup and plan on restoring that backup if you need to rollback to your previous level. Or, use alt_disk_install or multibos as a way to get back to your previous level. Technology Level updates should always be committed because they cannot be rejected. Committing the updates saves space in the / and /usr file systems and also makes it easier to track and reject service pack PTFs.
After the TL has been successfully applied and tested, another backup should be taken for disaster recovery situations.
Recommendation to apply the whole SP instead of just individual PTFs
For planned maintenance windows, customers are encouraged to apply Service Packs as a group, to simplify inventory and make it easier to report levels to service representatives and auditors. Fix Central and SUMA will now download the entire SP for a specified APAR (search function).
This was also done to simplify the ordering process.
For more information on Fix Central changes, see the paper:
Fix Central changes supporting AIX Service Strategy at
http://www14.software.ibm.com/webapp/set2/sas/f/best/home.html.
Even though the entire SP is downloaded, individual fileset updates or APARs can still be applied with SMIT (‘smitty install_by_fix’) or from the command line (installp or instfix). There is nothing in the Service Pack that glues all the updates in a Service Pack together. While
most of the testing that occurs in the regression test lab is done as a group, applying individual updates is still fully supported. For unplanned application of fixes (like security or HIPERs), changing the minimum amount of code is certainly the desired outcome.
For example:
• An interim fix was just released for a security issue and it requires an update from a
service pack.
• The service pack would require a reboot of the system whereas applying just the
individual PTF would not.
In these cases, just applying the individual APARs/PTFs may be the best solution. Service Packs, because they will always require a reboot, are recommended for planned maintenance windows or if the application of the fix will require a reboot anyway.
Service Pack schedules
Because there will be more Technology Levels shipping new fixes and because the Service Packs will be lined up to come out at the same time (with the same YYWW), the SP release schedules will be increased to approximately every 12 weeks. Occasionally, if a very critical problem is found (high severity security, for example) and it cannot wait for the next SP, a ‘standalone PTF’ will be released and will be made available via APAR search on Fix Central.
The PTF will be included in the next Service Pack.
Technology Level Lifecycle and Support With a valid Software Maintenance Agreement (SWMA), customers can continue to open PMRs to ask questions and report problems with AIX 5.2, AIX 5.3 and AIX 6.1. However, if a code defect is identified (i.e. an APAR is created) then the release and TL must be in the ‘New Fixes Available’ period to request a fix. This applies to both interim fixes and Service Pack updates.
More information about the System p Product Lifecycle dates can be found at:
http://www.ibm.com/software/support/systemsp/lifecycle
AIX Service Strategy Details and Best Practices
New Fixes Available – NOTE: All future dates are subject to change.
* New Fixes will be made available until two years later or when the new TL is offered. In some cases this may
mean slightly less than two years of support, in some cases it may mean slightly more. AIX will only make new
fixes available for up to four Technology Levels for each release at any given time.
** For 5.2, the date for ‘Fixes Available Until’ and End of Support are the same date.
When selecting a TL for upgrade, keep the ‘New Fixes Available’ period in mind. The
recommendation is to move to the latest TL and the latest SP when performing an upgrade, to get
the longest period of support.
Interim Fix Guidelines/Policy for Maintenance
Interim fixes are fixes made available to our customers to offer relief for a problem until the
customer is able to update to a Service Pack or Technology level that contains the fix. The
general rule is that a customer can request an interim fix for Technology Levels that are in the
‘New Fixes Available’ period; it will then be evaluated by development. Development will
make every attempt to supply an interim fix, however, there may be some that cannot be supplied
due to complexity.
Here are the guidelines that the Service team will be following:
• If a client is at the latest level and finds a problem – they can request an Interim Fix
• If a client finds a problem that is already in a later SP – they will be asked to install the
latest SP (or, at a minimum, the Service Pack that includes their fix or individual PTF that
contains the fix)
• If it is a critical problem and/or the client can’t install a SP – an interim fix can be
requested, however it is recommend they move to the latest SP during their next
maintenance window
• If a client finds a new, unique problem at any level that is making new fixes available -
they can get an Interim Fix on their current level (if it’s possible)
• If a client finds a new unique problem on a prior SP and has multiple Interim Fixes
installed, which are included in a later SP – they must move to the latest SP before getting
another Interim Fix
• If a client has multiple Interim Fixes and only some of them are in a later SP – they may
be asked to move to the latest SP and another Interim Fix will be bundled with their prior
Interim Fixes on the latest SP
• Clients should update to the SP that contains the fix as soon as they are able
Certified Interim Fixes
Interim fixes are offered to customers as a faster way to get temporary relief for a problem.
Interim fixes are shipped in ‘emgr’ format instead of installp format. Interim fixes are also
tracked via the ‘lslpp –L’ and ‘emgr –l’ commands.
Some interim fixes get additional functional and regression testing before they are released.
These interim fixes are usually intended for a wider audience than other interim fixes, which are
specific to a single customer.
Examples of certified Interim Fixes are security fixes that are released thru a vulnerability
advisory or subscription notification.
Security Fixes
Security fixes for security vulnerabilities are published thru advisories. Customers will continue
to be notified of security fixes by notices from IBM Subscription services at:
http://www14.software.ibm.com/webapp/set2/subscriptions/pqvcmjd
The security fixes published in the vulnerability advisories are posted here:
ftp://aix.software.ibm.com/aix/efixes/security
In the past, security interim fixes were offered for a longer period than other fixes to allow
customers time to transition to the current levels. But, because the enhanced service strategy
allows new fix availability for a longer period (even longer than the previous security fix policy
of 3 total TLs), security fixes will be made available for the same time period as all other new
fixes (see chart above). This offers consistency to the service strategy in regards to types of fixes
offered in the New Fixes Available period.
To assist our customers with the transition to AIX 5.3 Technology Levels that support the
enhanced service strategy, we will continue to generate 5.3 TL5 security interim fixes until 5.3
TL9 is shipped in 2H08 (approximately 10/2008). AIX 5.2 TL8 and AIX 5.2 TL9 support of
security interim fixes will continue until 6/2008. AIX 5.2 TL10 will continue to be supported
via Service Packs and interim fixes until end of support in April, 2009.
APAR Numbers for each TL
In the past, if a problem existed in two releases, only one APAR number was created for each
release. Starting with AIX 5.3 TL7, a new APAR number will be created for every Technology
Level where the problem is fixed. When PTFs were only made available on the current TL, only
one APAR number was required. Now that multiple service streams exist per release, an APAR
must be created for each one where a new fileset update (PTF) will be shipped.
If you’ve been given an APAR number for a particular problem, do a ‘Fix Search’ in Fix Central
to verify the APAR number with the Technology Level. The search will return results for the
other ‘sister’ APAR numbers, APAR numbers for the same release (e.g. 5.3) but different TLs,
and also ‘cousin’ APARs for other releases (e.g. 6.1). The search will also show availability of
the APAR and allow signup for a subscription notice if it has not been released.
For more information on searching for APAR and fixes, see the Fix Central paper at:
http://www14.software.ibm.com/webapp/set2/sas/f/best/home.html.
New Automatic Interim Fix Removal Function
Because of the issues with multiple APARs numbers in a release (APARs differ per TL as
described above) installp was not able to map the fixes it was shipping to the interim fixes that
were installed. Starting with 5.3 TL10 and 6.1 TL3, installp and emgr have been enhanced to
map the APARs and automatically remove interim fixes if the fix is present in the TL, SP or PTF
that installp is applying.
Interim fixes will now be packaged with a reference number. That number will allow installp to
map it back to APARs for all the Technology Levels where the fix was shipped.
When installp performs a preview install/update or an actual update, information will be
provided to list any interim fixes that will be automatically removed. Preview installs are always
highly recommended, even if you don’t currently have any interim fixes applied. But, if you do,
AIX Service Strategy Details and Best Practices
06/10/2009 10
then installp will list any interim fixes that are installed and that will be automatically removed
or any interim fixes that are installed and cannot be removed because the corresponding fix was
not available in the updates you were installing. In this later case, the install will fail stating
there are filesets locked by emgr, like it has in the past.
For example, the information about interim fixes that will be removed will appear directly after
the build date verification in the installp (apply or preview) output:
+—————————————————————————–+
BUILDDATE Verification …
+—————————————————————————–+
Verifying build dates…done
The updates being installed contain the following interim fix labels,
which will be removed prior to installing the updates:
IZ39706
IZ23965
NOTE: Rejecting an update that contains an interim fix will not
re-install the interim fix on the system.
……
Please note that if an interim fix is removed as part of the installp update, and then that update is
later removed (rejected), the interim fix that was previously installed will NOT be re-installed.
You would need to manually install the interim fix again at that point. So, always make sure you
have a copy of your interim fix, use alt_disk_install or create a mksysb image before the update
for recovery.
Because this is new function, a decision was made to not ship it in Service Packs on the ‘older’
Technology Levels. But, for this particular case, we will allow and support installing one
individual fileset from a TL. The bos.rte.install 5.3.10.0 update may be installed on 5.3 TL7,
TL8 or TL9 by itself, and the bos.rte.install 6.1.3.0 update may be installed on TL1 or TL2 by
itself. This will give you the new automatic interim fix removal function but will not require you
to go all the way to 5.3 TL10 or 6.1 TL3.
You may also download the bos.rte.install fileset update individually from Fix Central by
searching on bos.rte.install.
From the http://www-933.ibm.com/support/fixcentral/ main site, choose:
Product Group – System p, Product – AIX, Version – 5.3, Fix Type – Fix Search
http://www14.software.ibm.com/webapp/set2/fixsrch/FixSearch?release=53
Enter search items: fileset bos.rte.install
Then, in your search results you should select “Fileset information for: bos.rte.install”.
You will be taken to all the bos.rte.install fixes. Choose the latest update for your release so
you’ll get any additional fixes or function included. All the updates are cumulative, so you can
apply to any level since GOLD. You will need to uncheck the download options of ‘Include
requisites’, ‘Include fixes that correct regressions’ and ‘Replace superseded fixes with the latest’.
If you leave these options checked, then the entire TL will be listed for download.
It is also recommended to watch the bos.rte.install fileset to pick up any fixes or new function.
You can always update the bos.rte.install fileset to a lever higher than your current TL, so if new
function ships in a later TL, you can obtain it without updating your entire system.
All interim fixes shipping will have a reference field that allows installp to map the APARs to
the interim fix. When the interim fix is installed (if you have the new bos.rte.install fileset
applied) then a message will be printed by emgr to alert you to the fact that automatic removal is
enabled:
+—————————————————————————–+
Processing APAR reference file
+—————————————————————————–+
ATTENTION: Interim fix is enabled for automatic removal by installp.
To verify that your interim fix has a reference number, you can check the contents of the
/usr/emgrdata/DBS/aparref.db file:
# cat /usr/emgrdata/DBS/aparref.db
IZ39706|:|220
This file contains a list of interim fix labels with their corresponding reference number values.
Later, we’ll update the emgr –l command to show this reference number. So, again, it’s a good
idea to get the latest level of bos.rte.install to make sure you have all the latest fixes and
enhancements.
The bos.rte.install 5.3.10.0 or bos.rte.install 6.1.3.0 or higher update must be installed in order to
get the automatic interim fix removal function. If this fileset update is installed along with other
PTFs, in a Technology Level, for example, it will automatically re-invoke itself and it will then
perform the mapping and automatic removal, if applicable.
Hardware Support with a Service Pack
When new hardware is released, the required TL’s or SP’s will be published in the release notes
and RFA (Release For Announce). If the TL is still in the ‘New Fixes Available’ period, then
the latest SP will have the hardware support.
If new boot media is required for an existing TL, then it will be made available for web
download (ISO image) via the PRPQ process.
If you currently have the TL in your environment, then updating to the required SP and creating
a mksysb backup media or image (NIM) will allow boot and install of the new hardware
systems.
Moving to a New TL
You should move to a new TL:
• If your existing TL is out or is about to go out of the New Fixes Available period
• You want to use new function and/or features in a new TL. Hardware exploitation, such
as large page space or new software function, such as multibos, will only be released in a
TL.
• You are going to test a new level for distribution into your production environment and
want to get the longest period of new fixes available. In this case, you should move to
the latest TL.
Staying Current
If you are currently running on a TL where new fixes are being made available, then any SP will
be supported for your interim fix needs. But, the risk in not updating to a service pack that
contains your fix during a planned maintenance window is that you will require another interim
fix to the same file. See the Interim Fix Guidelines for more information.
Updating twice a year to a new SP or TL is recommended to stay current.
Maximum Stability Model
Realizing that customers appreciated and relied on the CSP for planning their maintenance, and
realizing that having a specific SP to recommend to customers that want to maintain the most
stable environment and do not plan on moving up to a new SP before moving to a new TL, we
would recommend using the SP that roughly would be delivered in the same time frame as the
CSP. We would recommend planning around SP3.
This is not to say that other Service Packs in the TL are less stable than SP3, but because the TL
has been out a few months and any major issues have usually been identified by that point in
time, and there is still more than a year left for service, SP3 is a good one to target and our
recommendation for maximum stability. This recommendation is meant as guidance for
planning. Individual situations may require applying earlier or later SP’s, depending on
circumstances.
Additional Information
The IBM Service and Support Best Practices for UNIX® servers Web page at
http://www14.software.ibm.com/webapp/set2/sas/f/best includes additional information on the
AIX Release and Service Strategy, and best practices for managing the AIX and System p™
servers.
The Fix Level Recommendation Tool (FLRT) is a planning tool to help administrators
determine what key components of your System p server are at the minimum recommended fix
level. It can be found at: http://www14.software.ibm.com/webapp/set2/flrt/home.
Summary
The new AIX service strategy will offer customers more options for maintenance in their
environment. Two years of new fixes per TL will help simplify and reduce cost for maintenance.
Supporting new hardware on existing TL’s will reduce complexity in customer environments.
« New FLRT upgrade includes Linux customers | Home | IBM Director 6.1 – Management tool to reduce TCO »
Leave a Comment