Data Product Dashboard API v0.9.0

Summary

Added

• Introduced a new endpoint: /ingestnewmetadata (POST). This endpoint allows you to ingest data product metadata directly through the REST API. Send a POST request with the contents of your metadata file formatted as JSON. The API will parse the JSON data and add the corresponding data product to the metadata store in use.
• Added a class to connect to an instance of PostgreSQL for use as a perisistant metadata store.
• Updated the PostgreSQL persistent metadata store to ingest all the metadata into the database when the application loads or new data products are added with the /ingestnewmetadata endpoint.
• Added SDP_DATAPRODUCT_API_ELASTIC_INDICES to environment variables to enable specification of Elasticsearch instances in deployments.
• Added DataProductMetadata class that contains methods related to handling of metadata in the application.
• BREAKING Added the concept of access_group to the metadata. This limits the access to data products if the user, authenticated with MS Entra, has not been assigned to the access_group of the data product. When using the API with the Data Product Dashboard, the user can authenticate with MS Entra. When loading the data products (using the filterdataproducts endpoint) the users access token will be used to retrieve the users assigned user groups, and that will be used as access list to determine which data products the user have access to.
    Data products that does not have an access_group assigned, will be open access to all users.
    Note: When using the API for scripted access to data products with the dataproductsearch endpoint, only data products with open access will be accessible.

Changed

• BREAKING This update refactors the data structure used to serve data to the MUI DataGrid component. It now aligns with the structure expected by the MUI DataGrid itself. This brings several improvements:
• • You can now leverage built-in MUI DataGrid features like column filters.
  • Full API Configurability: The table can be fully configured from the API, allowing for more granular control over its behaviours.
• Expanded the API status endpoint to include more information.
• Improved the git repository structure.
• Addition of basic authentication and self signed CA certificate for Elasticsearch developer environment.
• Updated make file to include options to create a development PostgreSQL and Elasticsearch Docker image .
• Restructured the data store to allow better integration between modes of operation.
• Updated the Elasticsearch query_body to be dynamically created based on search criteria from either the search parameters supplied from the dataproductsearch endpoint, the MUI DataGrid filter model or the Data Product Dashboard search panel.
• Updated the /dataproductsearch endpoint to enable search in both the in memory or Elasticsearch modes of operation.

•  BREAKING Changed the body or API request to specify the execution_block ID instead of fileName and relativePathName.
• The application now loads metadata either from a specified volume or when submitted to the API endpoints then saves it into either a persistent store using PostgreSQL or in memory if the database is not available. The search store then loads the metadata from either of these metadata stores into an Elasticsearch instance or an in-memory search store instance if Elasticsearch is not available.
• Updated project structure to align with the component functions, separating metadata stores and search stores.
• Moved passwords out of the env file into a .secrets file that is excluded from the repository and if found locally, loaded into environment variables.
• Updated PostgreSQL connection to enable connection to an existing DB instance in a cluster.
• Updated the configuration of the PostgreSQL metadata store to allow specification of a schema.
• The Elasticsearch http CA certificate needs to be loaded when deployed from the vault. To maintain the correct formatting, the certificate needs to be Base64 encoded before it is saved in the vault. The DPD API will now load the certificate from the vault into an environment variable, then decode it and save it in the format required for Elasticsearch. *
  Changed the Elasticsearch schema date_created field to date, and updated the query_body size to 100.
• BREAKING Integrated the API with the SKA Permissions API to enable it to obtain the users assigned user groups from MS Entra. *
  To enable better handling of connections and re-connections, a common PostgreSQL query method is added that returns the cursor object. When an operation fails due to a connection failure, it will retry and set the PostgreSQL availability to false on repeated failure.

 
Deprecated

• The in-memory store is deprecated and will be removed after all users have access to persistent PostgreSQL deployments. The functionality to load metadata into memory will be maintained to allow all users to configure and migrate to a persistent PostgreSQL DB for metadata storage.

Removed

• Removed /dataproductlist endpoint. This functionality has been replaced with either the /filterdataproducts which is aligned with the Data Product Dashboard requirements, or the /dataproductsearch endpoint that is a simplified version allowing for search and list of data products.

Fixed

• Fixed failure to load metadata into store when running the API with a PostgreSQL datastore.

Test Evidence

• Improved test coverage

Security

• None