Why archive SAP Data?

The usage of SAP systems results in the creation of huge amounts of data. Table sizes and storage needs grow exponentially.  The rapidly growing database starts to concern the IT department and archiving is brought up. Despite complaints about longer waiting times, the business usually resists the idea. Also, investment to a data archiving process is fairly hard to justify, why the first solution is probably to buy more disc. In the long run this is not a viable solution.

System Performance?

Archiving reduces the size of the database and prevents the growth. The goal is to lower the database costs and improve response times. SAP recommends to include the archiving already in the initial implementation plan. Decisions this stage can affect later archiving. For example, if you want to save the export invoices for a longer period than the others, it makes sense to create own billing type for these.

Version upgrade, conversions?

At some point a version upgrade becomes actual. SAP recommends archiving before upgrading in order to reduce the volume of transaction data. This results to shorter runtimes and production downtime. I have once participated in an upgrade that had to be interrupted. Because of the size of the database a weekend was too short.

Considering HANA?

It pays off to archive before a HANA Project. Although HANA is rapid and can handle tons of data, it does not mean that you should move old, useless data there. First of all, HANA is expensive. Memory is much more expensive than disc space.  SAP charges for HANA based on volume. A smaller database incurs lower hardware costs and helps speed up the migration process. Less documents to convert also enables shorter timelines for the project, from testing to the live conversion.

Legal Compliance – GDPR?

EU General Data Protection Regulation strengthens the data protection for all individuals within the European Union. Personal data must be deleted after the primary purpose of the processing has ended unless it has to kept because of other legislation – such as tax legislation. Access to it must be blocked or restricted, and it must be kept only for the duration of the longest legal retention period, after which it must be deleted as stated in GDPR.

To help with this task, as of SAP NetWeaver 7.40, SAP Business Suite applications provide blocking and deletion functionality that is based on SAP Information Lifecycle Management (SAP ILM).

Data archiving is the cornerstone of SAP ILM.

Phases of Archiving Project

SAP archiving involves complicated technical and business issues. To facilitate the communication between IT and business a project organization is best choice for the implementation. Project organization enables a better understanding and faster results.

Preparation

Archiving is not solely an IT project. It affects the business processes and user’s work. There are a lot of laws and regulations that govern the corporate information. These vary in different countries. The compliance to these is the responsibility of the business.

Although it is a common project, it pays off to prepare the project inside IT before kicking off and involving the business. They are not interested in the technical details, they are only interested in what will happen to their data.

Initial analysis (IT)

SAP archiving is probably novelty also for IT, why it is a good idea to engage an archiving consultant as a couch.

Identify and analyze:

• biggest tables and their growth rate
• archiving objects for these tables
• methods for data avoidance
• dependencies between archiving objects in business processes
• technical and business requirements
• laws and regulations
• access and reporting requirements

Identifying the biggest tables and their growth rate

As the main goal is to reduce the size if the database, the first thing to do is to find out what is in there. Which are the biggest tables and fastest growing tables? For this you have tools like SAP EWA reports, Data Volume Management (DVM) in Solution Manager or transaction DB02 or DB02OLD.

Which the biggest tables are depends on the business activities. This example below is from a consumer products company with heavy use of sales and logistics.

Quick Wins

Among the biggest tables, you will find huge technical tables containing spool data, application and change logs etc. In the above example table TST03 contains spool data. These are easy targets for archiving or deleting, because they do not contain data important to business. Also watch that your service provider regularly empties the spools and deletes old background jobs.

Data avoidance

Instead of collecting data, it is also possible to avoid it. Before deciding to archive you should analyze, whether there are any possibilities for this.

 

1. Data prevention — Stop the creation of unnecessary data

It pays off to explore, whether tables you don’t need, gather data. SAP Archiving guides (Data Management Guide for SAP Business Suite) and notes help to identify them. It is also possible to deactivate some technical parameters, which generate unwanted and unnecessary volume.

2. Data aggregation — Reduce unnecessary details by summarizing certain data

SAP offers different summarization options. Aggregates reduce the data volume. For example, FI document summary allows you to summarize the material from the data coming from MM and SD. Without the summarization the accounting document could easily have hundreds of lines.

3. Data deletion — Eliminate any data that is not necessary for business or legal purposes

There are transactions for deleting (reorganizing) administrative and technical data from SAP. If you want to delete application data or master data, you soon find out that it cannot be done.  It is done with archiving. But does it really make sense to store useless data like purchase requisitions that never matured to purchase orders?

When you implement ILM, you can also use archiving to clean old and unnecessary data. There is a new option Data Destruction, which deletes the archiving files.

Where to store the archived data?

The archived data is saved in a compressed form. The files are saved in the SAP file system, which is not a safe place to store important archives. The archive files are moved from SAP to external long-term archive. For this purpose, an archiving system needs to be procured. There are several vendors and alternatives for SAP certified archive systems.

Before project start you need to have the decision about the storage. It is important that the entire solution is at place, when you start the archiving tests. External archive systems are not cheap, which can be a problem.

Implementation partner

Most likely you want engage an implementation partner in the project. It is important to choose a partner that really knows archiving. It can be difficult to find a partner with comprehensive knowledge. Often the offerings concentrate just to the technical aspects of archiving and the knowledge of business requirements can be narrow.

 

Project Team

When you are clear with the preparations and have a vision of your project, it is time to prepare business for the coming and kick off the project.

Keep in mind that it is a business project, not an IT Project. Business owns the data.

IT is responsible for technical configuration of archiving objects, archiving info structures, archiving order and other technical areas. The main contribution of business is to define the retention periods both in SAP and the archive system. This information is vital for application specific configuration of archiving objects.

Data Analysis

In this phase you take a closer look at the data. Focus on largest ERP tables. Depending on sap version and the business, the largest tables vary, but very often you find these tables among the ten biggest.

• GLPCA – profit center line items
• FAGLFLEXA – new general ledger
• BSIS – open items
• BKPF, BSEG – accounting document
• MKPF, MSEG – material document
• COEP – controlling documents
• VBAK, VBRK, LIKP – SD order, deliveries, billing documents

There usually one archiving object for each table. The archiving object specifies precisely which data is archived and how. Transaction DB15 enables you to determine the archiving objects. Below the archiving object for SD Billing documents.

Dependencies

SAP archiving has lots of controls that ensure that data is fit for archiving is archived in right time and right order. For example, it is not possible to archive accounting documents that contain open items. A production order cannot be removed before the status is ‘Closed’. A sales order cannot be archived before all related deliveries documents are archived. Archiving validates that the business process is complete. If it is not, archiving fails. The reason can be faulty configuration or missing user action.

Take for example billing. It is not an isolated business transaction as you can see in the SD document flow.

There is an archiving object for each phase in the document flow.

There is also a defined sequence that must be followed when archiving these documents. Luckily you don’t have to know it, the archiving objects will check that their predecessors are archived before them.

Archiving master data (material, customers, and vendors) can be very painful. Prerequisite for archiving is that all transactions related to them has been archived. A material in an industrial company can have huge amount of dependencies from purchasing, production,  maintenance and sales.

Conceptual Design

One of the project results should be an archiving handbook, containing instructions and documentation of the process with all details. Start the document before testing and complement it during it. The archiving hand books should not remain a dead document on the shelf.

You need to create a test plan and implementation plan.

Continuous archiving

Archiving should be part of normal system maintenance and not firefighting. You need to plan:

• who takes care of the monthly, weekly, yearly archiving runs and who takes care of the archives?
• who checks the archiving logs to get the best result of the archiving? Who detects the erroneous ways of working that lead to incomplete processes and unarchivable data? Who informs business about faulty postings?
• who takes care of new archiving needs? For example, implementation of new modules and functionalities or organization and process changes that influence archiving?

Testing

Archiving requires thorough testing. End-users should be key resources in the test phase. Testing is an essential part of data archiving project.

At this stage you should also be able to show the users how to access the archived data. For some archiving objects the archived data is displayed with same transactions as unarchived. This is the case in accounting and purchase documents. Some applications have own transactions for archived data (e.g. Display archived invoice).  All participants should understand and accept the consequences of archiving in production before the decision is made.

Archiving must be planned and done correctly from the beginning. The knowledge grows with experience. You cannot test and validate too much. It is too late to regret, when the data is archived. How to explain the auditor that he cannot get the invoices, because you screwed up the archiving.

The initial tests often reveal that archiving is not going to be as simple as expected. In the SAP implementation all kinds of mistakes have been made. Accounts had been defined for ”open items accounts” but were never cleared.  Sales document flows are not completed. There are open returns and open deliveries.  This data cannot not be archived. The sales document flows can be so complicated that after tackling one error, tens of new errors appear.

The cause of these errors is that the users did something wrong and must be informed and trainred to avoid them in the future. Afterwards it is very difficult to improve the quality of the data and there is no motivation to correct old documents. This is often also impossible.

SAP has some corrections programs that can be used to force the documents to” complete”. However, these are not easy to find.

 

Don’t get greedy and try to handle too many archiving objects at a time. Divide the project in smaller parts.

Archiving runs can be very heavy and system intensive. To get a picture of the archiving times you need also to test the performance. However, remember that test and production systems do not have same load. Usually you should schedule the archiving runs to times with low system activity (nights, weekends).

Archiving does not automatically improve the response times. The freed space becomes available after reorganization of the archived tables and indexes. It is IT’s responsibility to take care of this.

The results of the project

The main result of a successful archiving project is the first archiving in the production. It is often common that the first archiving is clean-up archiving, which removes a lot of old data.

If you leave the archiving to this, after some time you find yourself in the same situation starting a new archiving project, where most of the work must be redone. For this reason, one of the project result should be a plan for continuous archiving.

Experiences

Theory is often different from practice. Technically archiving is pretty simple, but the process is not. There are lots of obstacles on the way.

Continuous archiving is a big challenge. When you search for resources, a vacuum is quickly formed around you. Also, many service providers are reluctant to offer archiving service. The unfortunate outcome is that someone from the IT department does the archiving on their own time. Some archiving runs can be automated using process chains, but also this requires supervision.

It can be very difficult to sell the project to business. Not only does it waste the valuable time of users, but also needs expensive equipment. When the work starts, there are usually a lot of ‘no-shows’ that have more important things to do.

Explain the consequences for not archiving. What it means both for performance and maintenance costs.

GDPR forces a new approach to archiving. It is no longer allowed to hoard decades old data only to be on the safe side. Business complete transaction data no longer needed in normal business operations should be archived and blocked. It is the responsibility of business to determine the retention and destruction strategies.