Understanding data import

  • Last update on December 12th, 2023

CoreView imports data from your Microsoft 365 tenant at different times: full and delta import.

Ways to import data from Microsoft

Microsoft offers various methods to import data from their environment:

Microsoft Graph

The Microsoft Graph REST API is a comprehensive endpoint that enables developers to access and interact with a diverse range of Microsoft services, including Microsoft 365, Azure Active Directory, OneDrive, SharePoint, Outlook, Teams, and others, through RESTful API calls. The Graph API offers the capability for real-time updates, allowing developers to receive notifications for changes happening in the data.

 
 

Microsoft Services API

Microsoft also provides APIs that can be accessed directly, without using PowerShell. These APIs are typically referred to by specific names based on the service or functionality they provide, such as Outlook API, Microsoft Teams API, SharePoint API, and more. They enable developers to programmatically interact with the corresponding services. These APIs are commonly RESTful APIs, accessible through HTTPS requests.

 
 

PowerShell with service accounts

Microsoft provides PowerShell modules and cmdlets that allow programmatic access to perform administrative tasks and operations within Microsoft 365.

 
 

How the data import works

CoreView retrieves various data from Microsoft 365 via API, such as user information, groups, and licenses from various providers: Azure AD, Teams, SharePoint, and more. 

Throughout the day, we perform delta imports to update specific properties whenever changes occur. Essentially, we continuously request information from Microsoft to check for any updates, and in case changes have indeed occurred, Microsoft notifies CoreView accordingly.

However, to ensure complete alignment between Microsoft and CoreView, a full import takes place once a day. This comprehensive import guarantees that all data within the tenant remains synchronized and accurate.

Full and delta imports, have their own unique characteristics and timetable:

Delta import

Thanks to Microsoft Graph, Microsoft has implemented the mechanism of “delta changes” where only values related to specific properties are updated during an import in near real-time. For example:

Property Values
userPrincipalName  user1@domain.com, user2@domain.com, user3@domain.com, …

However, there are other properties that do not adhere to this mechanism, and their values are only updated during a full import.

  • A delta import is only performed if values associated with properties that adhere to the “delta changes” mechanism have been changed or updated since the last successful delta import.
  • This import is performed every 5-10 minutes. providing a near real-time update. For example, if you create a new group from the Microsoft 365 Admin portal, that group will be available in CoreView after 5-10 minutes.
  • We cannot guarantee the duration it takes to complete. The time it takes for a full import may be subject to any data throttling Microsoft may have in place at that time. 
  • This import utilizes only Microsoft Graph.

If the tenant registered into CoreView platform has not been granted access to CoreView API Integration application, delta import is not available.

 

 List of properties that adhere to delta changes

The following Microsoft 365 properties are those whose values are updated when a partial import is performed. A partial import will only be performed if values associated with any of the following properties changed/updated since the last successful partial import. 

Azure AD

Properties whose values are updated for online users when a partial import is performed.

accountEnabled 
assignedLicenses 
assignedPlans 
businessPhones 
city * 
companyName * 
country * 
createdDateTime 
customAttributes 
department * 
displayName 
employeeId 
givenName 
id 
imAddresses 
jobTitle 
mail 
mailNickname 
mobilePhoneonPremisesLastSyncDateTime  
onPremisesSyncEnabled  
postalCode * 
preferredLanguage  
proxyAddresses  
recipientType (retrieved from Exchange)  
recipientTypeDetails (retrieved from Exchange)  
state * 
streetAddress  
surname  
usageLocation * 
userPrincipalName  
provisionedPlans  
officeLocation  
onPremisesDistinguishedName  
CustomAttributes (onPremisesExtensionAttributes)  
onPremisesImmutableId  
onPremisesSamAccountName

 
 

Licenses

Newly acquired licenses are updated during partial import so that if a new license is added, it will become available in near real time.

SkuId 
SkuPartNumber 
ConsumedUnits 
PrepaidUnits 
ServicePlans 

 
 

Groups

Properties whose values are updated for groups and members when a partial import is performed.

Description 
DisplayName 
Mail 
MailEnabled 
MailNickName 
OnPremisesLastSyncDateTime 
OnPremisesSecurityIdentifier 
OnPremisesSyncEnabled 
OnPremisesDomainName 
OnPremisesNetBiosName 
OnPremisesSamAccountName 
ProxyAddresses 
SecurityEnabled 
Visibility 
CreatedDateTime 
RenewedDateTime 
ResourceProvisioningOptions 
Members 
Owners **

 
 

For more information and details please check Microsoft documentation here: https://docs.microsoft.com/en-us/graph/api/resources/group?view=graph-rest-1.0 

* Properties available for use as a License Pool Filter Criteria , which is recalculated every 30 minutes. 

** In case only the owner is modified, then this change won't be captured by the partial import but imported with the full import. In addition to the owners, any other properties can be changed (add a member and remove it). This way the partial import will import the change. 

Some properties of entities or functionalities (users, groups, etc.) may adhere to the delta changes mechanism, while others may not. For example, when it comes to user information, some properties adhere to this mechanism while others do not.

 

Full import

  • The complete data is imported from all the providers, including SharePoint, Teams App, Teams Voice, Exchange, and more.
  • It utilizes Microsoft Graph, Microsoft Services API, and PowerShell with service accounts.
  • Performed once a day, approximately every twenty-four hours. In other words, the data is updated at least once within a twenty-four hour period.
  • This import is typically performed after regular working hours, taking into account Microsoft's data management practices. We strive to schedule the import during nighttime across different data centers, but the exact timing is ultimately determined by Microsoft data management practices.. Therefore, we cannot provide a specific time guarantee for when the import will run.
  • We cannot guarantee the duration it takes to complete. The time it takes for a full import may be subject to any data throttling Microsoft may have in place at that time. 
  • The import process requires a considerable amount of time, which is why it cannot run more than once a day.
  • There is no way to force a full import. 

During the full import, a data that has been imported during a delta import via Microsoft Graph will not be overridden by the data retrieved with different kind of APIs (with or without PowerShell).

 

Questions

Why can I see changes on CoreView but not in a specific environment, like Azure AD?

All the data on the Microsoft 365 Admin portal is exposed through the Graph API. Therefore, it is possible that if, for example, you create a new group on the Microsoft 365 portal, it will be visible in CoreView but not yet in AzureAD.

 
 

A property can exist on different providers. How can I determine the source of a specific property?

We know that the same property can exist on different providers. Based on experience and testing, we know that importing a specific property from one provider is preferable over another.

 
 

How does the import for Endpoint Management work?

Endpoint management only supports full imports at the moment. Unfortunately, Microsoft does not currently offer the capability for partial imports within Endpoint management.