About Brightspace Data Sets
Brightspace Data Sets contain raw, user-level data from Brightspace Learning Environment, and can be combined with other Advanced Data Sets and Brightspace Data Sets to provide administrators with the maximum flexibility in aggregating and filtering the data specific to their needs.
To track changes to Brightspace Data Sets in Brightspace Platform releases, Brightspace Data Sets contain a version number. For example, in the Brightspace Platform 20.19.05/May 2019 release, Brightspace Data Sets were updated to version 3.3. Specific changes to individual Brightspace Data Sets are captured in their respective topics. For example, if a new field is added to a data set, the topic includes the updated field and version number it was introduced in.
|Brightspace Platform Version||Brightspace Data Sets Version|
3.0 - Contains all changes
2.5 - Does not include new and deleted columns for Users data set
Version history comments
In addition to version numbers, each Brightspace Data Set topic includes a brief description of any field changes. For example:
|Version History Comment||Context|
|Added||The field has been added as a new column at the end of the data set.|
|Added Primary Key designation||The field has become a singleton or part of a composite primary key to uniquely identify rows in the data set.|
|Removed Primary Key designation||The field is no longer used as a singleton or part of a composite primary key in the data set. Unless otherwise specified, the field remains in the data set.|
Updating the version number
Beginning with the 10.7.5/September 2017 release, the Continuous Delivery (CD) Update process automatically updates the Brightspace Data Sets version number for releases containing low impact changes (non-breaking changes to downstream systems or reporting, such as adding a column). Normally, a major version change to Brightspace Data Sets is optional; however, when the change occurs to resolve a defect, clients are automatically updated. With the June 2019 release, the 6 month opt-in window to opt in to Brightspace Data Sets 3.x is over; clients are automatically updated. If you already use version 3, there is no change. For more information about managing Brightspace Data Sets updates, see the Updating The Brightspace Data Sets Change Management Policy on the Brightspace Community.
Note the following:
- Once the Brightspace Data Sets version number has been updated, changes are reflected the next time the Brightspace Data Sets are generated.
- If the Brightspace Data Sets are in the process of being generated when the version number is changed, any data set created after that change has been made reflects the latest version. So, it is possible that a subset of the data sets might reflect the previous version and a subset might reflect the latest version.
Full Data Sets and Differentials
To analyze and report on the data, Brightspace Data Sets are available in both full data sets and differentials. Full data sets are complete extracts of historical data, refreshed weekly. While full data sets contain a complete view of historical data, they result in large data extracts, taking longer to download. Differential (diff) data sets contain the differences (diffs) of the data that has been updated or inserted in the previous day, and refreshed daily. Daily diffs result in smaller and more up-to-date data extracts, making it faster for administrators to download and process the data.
Note the following about differential data sets:
- To prevent data gaps, updated data sets include some overlap. For example, an updated data set may contain rows from the last few minutes of the previous hour and the next few minutes of the next hour.
- Only inserts or updates display, including updates to existing Is_Deleted columns.
- Only updates on first-class data display. For example, the Quiz Attempts data set displays additional attempts; however, updates to the QuizName field do not display.
For administrators with high-frequency reporting needs, clients can purchase an add-on for the Data Hub tool that refreshes specific Brightspace Data Sets on an hourly basis. The add-on is available to AWS-hosted Brightspace Insights and Brightspace Core clients only.
Note the following exceptions about Brightspace Data Sets:
- The Content User Progress, Discussion Post Read Status, Discussion Posts, Quiz User Answer Responses, Quiz User Answers, Session History, User Logins data sets are limited to 3 years of data (all of the previous two calendar years and the current calendar year to date).
- The differentials for Quiz User Answer Responses, Session History, and User Logins are also limited to 3 years of data (all of the previous two calendar years and the current calendar year to date).
- All other Brightspace Data Sets adhere to the default system limit to return 150 million rows of the most recent data.
Scheduling generation of Brightspace Data Sets
For each Brightspace Data Set, differential data sets generate on a daily basis. In addition to the differential, the full CSV data set continues to generate once a week.
Hourly data freshness add-on
Clients can purchase an add-on for the Data Hub tool that generates specific Brightspace Data Sets on an hourly basis. This feature is intended for administrators with high-frequency reporting needs.
- For each Brightspace Data Set, differential data sets generate on an hourly basis, and capture any data inserted or updated in the last hour.
- In addition to the differential, the full CSV data set continues to generate once a day.
The add-on is available to AWS-hosted Brightspace Insights and Brightspace Core clients in the following regions:
- North America
To enable the add-on for the Data Hub tool, contact your D2L Account Manager.