Processing the data

Data collection

At regular intervals during the year agencies, such as KDE, CPE and EPSB, submit their data to KCEWS. We then collect the data and add it to the KLDS. KCEWS collects and holds data based on academic years. At the end of each year new data is processed and the submitting agencies get the opportunity to revise and update their submissions. The new data is then merged into the main data warehouse, regular reports are generated, and the Business Objects Universe is refreshed.

Data quality

Many of the uploaded files have fields that can be edited to ensure the data is valid, for example a school name or district name must match the defined names provided by the state. Certain fields must contain numbers, or be within a designated range. Dates must adhere to a specific format and range. The data import process will use a variety of table lookups and validation steps to ensure the highest possible quality of the data.

Data cleaning

Some of the data may be incomplete or inaccurate. In these cases the data impacted will be identified and placed into a folder for manual review and potential correction. If the data cannot be clearly and accurately updated, then the source agency of the data will be notified.

The processing cycle

The process of uploading, validating, cleaning and matching the data is a complex cycle of activities. These activities have been designed and developed by the KCEWS team over multiple iterations of building the data warehouse. We have included an audit and review process as the final step to ensure quality before the data is allowed to enter the final production environment.