Upload 184 files

files/arthur-docs-markdown/index.rst.txt

@@ -0,0 +1,75 @@
+Arthur Docs Home
+===========================================================
+
+`Arthur <https://arthur.ai>`_ is the platform for centralized monitoring of production models.
+We help data scientists, product owners, and business leaders accelerate model operations to optimize for accuracy,
+explainability, and fairness. As a model- and infrastructure-agnostic platform,
+Arthur adds a layer of intelligence to your AI stack and scales with your deployments.
+
+This documentation website is meant to help new users get started with Arthur, and to provide helpful
+guides to existing users so they can get the most out of their production models with Arthur.
+
+For complete references for the Arthur API or the Python SDK, see our :doc:`API & SDK References <api-references>`.
+
+.. container:: nav-card
+
+   :doc:`Quickstart <user-guide/arthur_quickstart>`
+
+.. container:: nav-card
+
+   :doc:`User Guide <user-guide/index>`
+
+.. container:: nav-card
+
+   :doc:`Platform Management <platform-management/index>`
+
+.. container:: nav-card
+
+   :doc:`API & SDK References <api-references>`
+
+*****
+
+========
+Contents
+========
+
+
+:doc:`Arthur User Guide <user-guide/index>`
+***********************************************************
+
+The Arthur User Guide contains everything you need to know to use the Arthur platform. If you're an Arthur end user,
+start here.
+
+:doc:`Platform Management <platform-management/index>`
+***********************************************************
+
+The Platform Management section contains information for Arthur product and infrastructure administrators, with guides
+for on-prem deployments and organization management.
+
+:doc:`More Information <more-info/index>`
+***********************************************************
+
+Additional details about algorithms used in the Arthur platform, frequently asked questions, and a glossary of Arthur
+terms.
+
+.. _API and SDK References:
+
+:doc:`API & SDK References <api-references>`
+***********************************************************
+See our `Arthur REST API Reference <https://docs.arthur.ai/api-documentation/v3-api-docs.html>`_ for a complete
+outline of the endpoints available in the Arthur API.
+
+See our `Arthur Python SDK Reference <https://docs.arthur.ai/sdk/sdk_v3/index.html>`_ for a complete outline of the
+functions available in the Arthur Python SDK.
+
+.. toctree::
+   :hidden:
+   :maxdepth: 2
+
+   self
+   user-guide/index
+   platform-management/index
+   more-info/index
+   api-references
+   Example Projects <https://github.com/arthur-ai/arthur-sandbox>
+

files/arthur-docs-markdown/internal/how-to-images.md.txt

@@ -0,0 +1,19 @@
+---
+orphan: true
+---
+
+# Markdown Syntax
+
+All image use the same syntax regarding of the file extension. The structure is `[alt text][image path]`
+
+## Jpg
+![.jpg image](https://uploads-ssl.webflow.com/5a749d2c4f343700013366d4/60772435a896b19234c5a7bb_maxresdefault-p-800.jpeg)
+
+## Gif
+![.gif image](https://uploads-ssl.webflow.com/5a749d2c4f343700013366d4/607481fec86baf2a8cb273d8_monitor.gif)
+
+# HTML syntax
+
+We can also use regular HTML image tags if we want to add styling to our image.
+
+<img src="https://uploads-ssl.webflow.com/5a749d2c4f343700013366d4/607481fec86baf2a8cb273d8_monitor.gif" alt="Alt text in case source link is broken" style="height: 200px; width: 200px; object-fit: contain" />

files/arthur-docs-markdown/more-info/FAQs.md.txt

@@ -0,0 +1,88 @@
+# FAQs
+
+**1. Can I use Arthur without using the Python SDK?**
+
+Yes! The Arthur platform is API-first. You can use our 
+[REST API](https://docs.arthur.ai/api-documentation/v3-api-docs.html) to onboard models, 
+send predictions, and query metrics and insights.
+***
+
+**2. Does Arthur need a copy of my model?**
+
+Arthur doesn’t generally need access to your actual model, but only captures the inputs to the model 
+and predictions it makes. This means that you can even use Arthur to monitor models you have no access to, 
+such as models hosted by third-party services.
+
+In order to enable explainability, Arthur _does_ need access to your model. 
+When {ref}`enabling this feature <enrichments_explainability>`, 
+you will need to provide access to the model's `predict` function.
+***
+
+**3. What if my data is proprietary? Can I still use Arthur?**
+
+Yes! Arthur offers on-premises installation for customers with data security requirements. By integrating Arthur
+into your business's on-premises stack, you can be confident that all security requirements are met while still
+getting the benefits of the computation and analytics Arthur provides.
+***
+
+**4. What if I don’t have ground truth labels for my data? Or what if I will have the ground truth labels in the future, 
+but they are not available yet?**
+
+You don't need ground truth labels to log your model's inferences with Arthur. 
+
+If your ground truth labels become available after your model's inferences, whether seconds later or years later, 
+Arthur can link these new ground truth values to your model's past predictions, linking the new values by ID to 
+their corresponding inferences already in the Arthur system.
+
+In the meantime, Arthur’s data drift metrics can offer leading indicators of model underperformance to keep you 
+covered if your ground truth labels are delayed or never become available.
+***
+
+**5. I got an error using the SDK. What do I do?**
+
+If the error message says "an internal exception occurred, please report to Arthur" that means there was a problem 
+on our side. Please email the Arthur support team at `support@arthur.ai` to let us know what happened. 
+
+Otherwise, the error message should provide helpful instructions for how to resolve the issue. If you don’t find 
+the error message actionable, please let Arthur know so that we can improve it.
+***
+
+**6. Do I have to type my credentials in every time I use the SDK?**
+
+No! Instead of manually entering them you can specify an `ARTHUR_ENDPOINT_URL` and `ARTHUR_API_KEY` 
+environment variable to be used to create the ArthurAI connection object.
+***
+
+**7. What are streaming and batch models?**
+
+{ref}`Streaming and batch models <basic_concepts_streaming_vs_batch>` are two model types with different 
+patterns of ingesting data to send to Arthur.
+
+A streaming model processes data as a stream of individual inferences: data is logged with Arthur directly as 
+individual inferences when the data flows into the model.
+
+A batch model processes data as a sequence of grouped inferences, which are usually grouped over time: data is 
+logged with Arthur as a group of inferences as the model processes the batch.
+***
+
+**8. Which drift metric should I use?**
+
+Population stability index (PSI) is typically a good default drift metric to use.
+
+There are some cases where one wants a drift metric with a certain property, e.g. using a drift metric with the unit 
+nats for interpretability, or using a drift metric bounded between 0 and 1 so that drift values don't increase 
+arbitrarily for outliers. In these cases, other metrics may be preferable to PSI. 
+
+For a review of the data drift metrics Arthur offers and their properties, see the 
+{ref}`data drift section of our glossary <glossary_data_drift>`. Furthermore, see 
+[our blog post](https://www.arthur.ai/blog/automating-data-drift-thresholding-in-machine-learning-systems) for an
+overview of data how Arthur automates the choice of thresholding for drift metrics.
+
+**9. Do I need to configure my attribute ranges in order to onboard my model?**
+
+No - none of Arthur's model performance metrics or monitoring capabilities are impacted by the `range` property of an 
+attribute. So, if you observe the attribute looks off when you are reviewing your model during onboarding, you should feel fine
+going ahead and saving your model to the platform.
+
+The `range` property of an attribute - the min/max values of numerical attributes - is only a measurement taken by 
+Arthur for making decisions on how to format plots in the online dashboard.

files/arthur-docs-markdown/more-info/algorithms.md.txt

@@ -0,0 +1,171 @@
+# Arthur Algorithms
+
+(arthur_algorithms_anomaly_detection)=
+## Anomaly Detection
+
+For an explanation of how to enable Anomaly Detection with Arthur, please see the {ref}`enrichments guide <enrichments_anomaly_detection>`.
+
+Anomaly scores are computed by training a model on the reference set you provide to Arthur, and using that model to assign
+an Anomaly Score to each inference you send to Arthur. Scores of 0.5 are given to "typical" examples from your reference
+set, while higher scores are given to more anomalous inferences and lower scores are given to instances that the model
+judges as similar to the reference data with high confidence.
+
+### How Anomaly Detection Works
+We calculate Anomaly Scores with an Isolation Forest algorithm. This algorithm works by building what is essentially a density model of the data by iteratively isolating data points from one another. Because anomalies tend to be farther away from other points and occur less frequently, they are easier to isolate from other points, so we can use a data point's "ease of isolation" to describe its anomaly. The method is based on the paper linked [here](https://cs.nju.edu.cn/zhouzh/zhouzh.files/publication/icdm08b.pdf?q=isolation-forest).
+
+The Isolation Forest "method takes advantage of two [quantitative properties of anomalies]:
+    i) they are the minority consisting of fewer instances and
+    ii) they have attribute-values that are very different from those of normal instances.
+In other words, anomalies are 'few and different', which make them more susceptible to isolation than normal points." 
+
+The idea is to build a binary tree which randomly segments the data until each instance can be uniquely selected (or a maximum height is reached). Anomalous instances will take fewer steps to become isolated on the tree, because of the properties mentioned above.
+
+![iso-forest-example](/_static/images/iso-forest-example.png "ISO Forest Example")
+
+In the example above, we can see that the in-distribution $x_i$ takes many more steps to reach than the out-of-distribution $x_0$
+
+Of course using a single randomly generated tree would be noisy. So we train multiple trees to construct an Isolation Forest of multiple trees and use the average path length, noting that average path lengths converge:
+
+![iso-forest-paths-converge](/_static/images/iso-forest-paths-converge.png "ISO Forest Path Convergence")
+
+The Isolation Forest algorithm is highly efficient compared to other density estimation techniques because the individual trees can be built from samples of the data without losing performance.
+
+When you add a reference set to your model in Arthur, we fit an Isolation Forest model to that data, so that we can compute an anomaly score for the inferences your model receives.
+
+### Generating Anomaly Scores
+
+The path length, or number of steps taken to reach the partition that a data point belongs to, varies between $0$ and $n-1$ where $n$ is the number of datapoints in the training set. Following our intuition above, the shorter the path length, the more anomalous a point is. In order to measure anomaly, an anomaly score between $0$ and $1$ is generated by normalizing the path length by the average path length and applying an inverse logarithmic scale.
+
+In particular, anomaly score $s(x,n) = 2^{-\frac{E(h(x))}{c(n)}}$, where $E(h(x))$ is the average path length to datapoint $x$from a collection of isolation trees and $c(n)$ is the average path length given the number of datapoints in the dataset $n$.
+
+Every inference you send to Arthur will be evaluated against the trained Isolation Forest model and given an anomaly score.
+
+This can be seen in the anomaly score contour of 64 normally distributed points:
+
+![iso-forest-contour](/_static/images/iso-forest-contour.png "ISO Forest Contours")
+
+At Arthur, we also visualize the distribution of anomaly scores among all of the inferences your model has retrieved since training. When you select an inference in our UI you'll see where it falls on this distribution:
+
+![anomaly-score-distribution](/_static/images/anomaly-score-distribution.png "Anomaly Score Distribution")
+
+### Interpreting Anomaly Scores
+
+The resulting anomaly scores can be interpreted in the following way:
+- if instances return $s$ very close to $1$, then they are definitely anomalies
+- if instances have $s$ smaller than $0.5$, then they are quite safe to be regarded as normal instances
+- if all the instances return $s \approx 0.5$, then the entire sample does not really have any distinct anomaly
+
+-----------
+(arthur_algorithms_bias_mitigation)=
+## Bias Mitigation
+
+If you are interested in mitigation capabilities, we're happy to have a conversation about your needs and what approaches would work best for you.
+Within the Arthur product, we offer postprocessing methods; we do encourage exploration of alternate (pre- or in- processsing) methods if your data science team has the bandwidth to do so. 
+
+We currently have one postprocessing method available for use in the product: the Threshold Mitigator.
+
+### Threshold Mitigator
+This algorithm is an extension of the Threshold Optimizer implemented in the `fairlearn` library, which in turn is based on Moritz Hardt's 2016 paper introducing the method. 
+
+The intuition behind this algorithm is based on the idea that if the underlying black-box is biased, then different groups have different predicted probability distributions — let's call the original predicted probability for some example $x$ the score $s_x$. Then, for a *fixed threshold* $t$, $P(s_x > t | x \in A) > P(s_x > t | x \in B)$, where $B$ indicates membership in the disadvantaged group. For example, the default threshold is $t = 0.5$, where we predict $\hat Y = 1$ if $s > 0.5$. 
+
+However, we might be able to **mitigate bias in the predictions by choosing group-specific thresholds for the decision rule.** 
+
+So, this algorithm generally proceeds as follows: 
+
+- Try every possible threshold $t$ from 0 to 1, where for any $x$, we predict $\hat Y = 1$ if and only if $s_x > t$.
+- At each of those thresholds, calculate the group-specific positivity, true positive, and false positive rates.
+
+Let's say we're trying to satisfy demographic parity. We want the group-wise *positivity rates* to be equal —  $P(\hat Y = 1 | A) = P(\hat Y  = 1 | B)$. Then, for any given *positivity rate*, the threshold $t_a$ that achieves this positivity rate might be different from the threshold $t_b$ that achieves this positivity rate.  12*
+
+This also hints at why we need the notion of *tradeoff curves*, rather than a single static threshold: if we want to achieve a positivity rate of $0.3$, we'll need different $t_a, t_b$ than if we wanted to achieve a positivity rate of $0.4$. 
+
+Then, once we've generated the curves, we pick a specific *point on the curve*, such as positivity rate of $0.3$. When making predictions on future  and use the corresponding $t_a, t_b$ to make predictions 
+
+#### What's Compatible?
+
+**Fairness constraints:** We can generate curves satisfying each of the following constraints (fairness metrics): 
+
+- Demographic Parity (equal positivity rate)
+- Equal Opportunity (equal true positive rate)
+- Equalized Odds (equal TPR and FPR)
+
+**Sensitive attributes:** This algorithm can handle sensitive attributes that take on any number of discrete values (i.e. is not limited to binary sensitive attributes). For continuous sensitive attributes, we can specify buckets for the continuous values, then treat them like categorical attributes. 
+
+#### The tradeoff curve
+
+**Generating the Curve** 
+
+- Each *group* has its own curve.
+- To generate a single curve, we generate all possible thresholds (i.e. `[0, 0.001, 0.002, 0.003, ..., 0.999, 1.00]`) — default 1000 total thresholds; as described above, for each of those thresholds, we calculate many common fairness metrics for the result *if* we were to use that threshold to determine predictions.
+
+**Visualizing the Curve**
+
+Depending on the fairness constraint we're trying to satisfy, the tradeoff curve will look different. 
+For equal opportunity and demographic parity, where there is a *single* quantity that must be equalized across the two groups, the tradeoff curve plots the quantity to equalize on the x-axis, and accuracy on the y-axis.
+
+<!-- ![https://s3-us-west-2.amazonaws.com/secure.notion-static.com/c9d0b91f-3201-4b93-b526-d898cd23257a/Untitled.png](https://s3-us-west-2.amazonaws.com/secure.notion-static.com/c9d0b91f-3201-4b93-b526-d898cd23257a/Untitled.png) -->
+
+**Understanding the Curve Artifact** 
+
+The curve artifact can be pulled down according to our docs [here](https://docs.arthur.ai/api-documentation/v3-api-docs.html#tag/bias-mitigation). This mitigation approach is somewhat constrained in the sense that we will generate a separate set of curves *for each sensitive attribute to mitigate on; for each constraint to follow.* Then, each *set* of curves entails a single curve for each sensitive attribute *value*. 
+
+- Conceptually, the curves are organized something like the below; in the API, however, you'll be getting a single `< list of (x, y) coordinate pairs that define the curve >` at a time, with additional information about the attribute being mitigated; the constraint being targeted; and the feature value the specific list of pairs is for.
+
+    ```
+    **mitigating on gender -- demographic parity**
+    	gender == male
+    		< list of (x, y) coordinate pairs that define the curve >
+    	gender == female
+    		< list of (x, y) coordinate pairs that define the curve >
+    	gender == nb
+    		< list of (x, y) coordinate pairs that define the curve >
+
+    **mitigating on gender -- equalized odds**
+    	gender == male
+    		< list of (x, y) coordinate pairs that define the curve >
+    	gender == female
+    		< list of (x, y) coordinate pairs that define the curve >
+    	gender == nb
+    		< list of (x, y) coordinate pairs that define the curve >
+
+    **mitigating on age -- demographic parity**
+    	age < 35
+    		< list of (x, y) coordinate pairs that define the curve >
+    	age >= 35
+    		< list of (x, y) coordinate pairs that define the curve >
+    ```
+
+**Summary of Curves by Constraint**
+
+- Equal opportunity (equal TPR): TPR vs accuracy; accuracy-maximizing solution is the point along the x-axis with the highest accuracy (y-axis) for both groups.
+- Demographic parity (equal selection rates): selection rate vs accuracy; accuracy-maximizing solution is the point along the x-axis with the highest accuracy (y-axis) for both groups.
+- Equalized odds (equal TPR *and* FPR): FPR vs TPR (canonical ROC curve); accuracy-maximizing solution is the point on the curves that are closest to the top left corner of the graph (i.e. low FPR and high TPR).
+
+#### Choosing a Set of Thresholds (Point on Curve)
+
+As mentioned above, a single "point" on the solution curve for a given constraint and sensitive attribute corresponds to several thresholds mapping feature value to threshold. But how do we choose which point on the curve to use? 
+
+The default behavior — and, in fact, the default in fairlearn — is to automatically choose the accuracy-maximizing thresholds subject to a particular fairness constraint. This might work in most cases; however, accuracy is not necessarily the only benchmark that an end-user will care about. One client, for example, needs to satisfy a particular positivity rate, and is fine with sacrificing accuracy to do so. What our feature introduces that goes beyond what's available in fairlearn is the ability to try different thresholds; see what hypothetical predictions would be; and see what hypothetical results/metrics (eg positivity rate, TPR, etc) would look like *if*  a certain set of thresholds was applied. Ultimate choice of thresholds is up to data scientists' needs, etc.
+
+---------
+(arthur_algorithms_hotspots)=
+## Hotspots
+
+
+For an explanation of how to enable Hotspots with Arthur, please see the {ref}`enrichments guide <enrichments_hotspots>`.
+
+We might have a ML model deployed in production and some monitoring in place. We might notice that performance is degrading from classic performance metrics or from drift monitoring combined with explainability techniques. We’ve identified that our model is failing, and the next step is to identify why our model is failing.
+
+This process would involve slicing and dicing our input data that caused model degradation. That is, we want to see which particular input regions are associated with poor performance and work on a solution from there, such as finding pipeline breaks or retraining our models on those regions.
+
+This basically boils down to a time-consuming task of finding needles in a haystack. What if we could reverse engineer the process and surface all of the needles, i.e. input regions associated with poor performance, directly to the user?
+
+We can with Hotspots!
+
+### How Hotspots Works
+The outputs of a model are encoded as a special classification task to partition data to separate out the poor performing datapoints from the correct predictions/classifications, on a per batch basis for batch models or on a per 7-day window for streaming models.
+
+Hotspot enrichments are used to surface input regions where the model is currently under performing on for inferences. Hotspots are extracted from a custom Arthur tree model, where nodes are associated with particular input regions and have associated performance metrics, e.g. a node with 70% accuracy that has datapoints where variable X is less than 1000. Nodes are candidates for hotspots. Depending on user-specified thresholds, e.g. a threshold of 71% accuracy, the tree is traversed until all nodes with less than 71%, such as our node with 70% accuracy, have been identified and returned to the user as hotspots, not including the hotspot nodes' children, which would be either (1) more pure than the hotspot node and therefore in further violation of the e.g. 71% threshold or (2) pure nodes with correct inferences, which are not of interest to the user for remediation purposes.
+
+For a more in depth overview, see our blog post [here](https://www.arthur.ai/blog/hotspots-automating-underperformance-regions-surfacing-in-machine-learning-systems).

files/arthur-docs-markdown/more-info/glossary.md.txt

@@ -0,0 +1,599 @@
+# Glossary
+
+The following definitions are specific to the Arthur platform, though in most cases are applicable to ML more broadly.
+
+## Arthur Inference<a name="arthur-inference"></a>
+Container class for inferences uploaded to the Arthur platform. An inference is composed of input features, prediction values, and (optionally) ground truth values and any Non-Input data.
+
+Example:
+```python
+ground_truth = {
+    "Consumer Credit Score": 652.0
+}
+inference = arthur_model.get_inference(external_id)
+inference.update(ground_truth)
+```
+
+Related terms: [inference](#inference-data), [ArthurModel](#arthur-model)
+
+## Arthur Model<a name="arthur-model"></a>
+Model object used for sending and retrieving data pertinent to a deployed ML system. The `ArthurModel` object is separate from the underlying model that is trained and which makes predictions; it serves as a wrapper for the underlying model to access Arthur platform functionality.
+
+An ArthurModel contains at least a `name`, an `InputType` and a `OutputType`.
+
+Examples:
+```python
+arthur_model = connection.model(name="New_Model",
+                               input_type=InputType.Tabular,
+                               model_type=OutputType.Regression)
+```
+
+```python
+arthur_model = connection.get(model_id)
+arthur_model.send_inference(...)
+```
+
+## Arthur Model Group<a name="arthur-model-group"></a>
+
+Arthur Model Groups are an organizational construct the Arthur platform uses to track different versions of an Arthur model. Every Arthur Model is a version of one Model Group and a Model Group will always have at least one Arthur Model. The Model Group for an Arthur Model can only be specified during on-boarding, and once the Arthur Model is saved, its group cannot be changed. If an Arthur Model is created without specifying a Model Group, a new Model Group will be created automatically with the new model as its single version. When adding a model to a model group, the model is assigned a unique, incrementing Version Sequence Number (starting at 1) corresponding to the order in which it was added to the model group. Additionally, you can provide a Version Label to store a custom version string label along with the Version Sequence Number.
+
+Example:
+
+```python
+# retrieve the first version of a model
+arthur_model_v1 = connection.get(model_id)
+model_group = arthur_model_v1.model_group
+
+# create the new version of the model
+arthur_model_v2 = connection.model(name="Model_V2",
+                                   input_type=InputType.Tabular,
+                                   model_type=OutputType.Regression)
+
+# add the new model to the model group
+model_group.add_version(arthur_model_v2, label="2.0.1")
+arthur_model_v2.save()
+```
+
+Related terms: [Version Label](#version-label), [Version Sequence Number](#version-sequence-number)
+
+## Attribute<a name="attribute"></a>
+A variable associated with a model. Can be input, prediction, ground truth or ancillary information (these groupings are known as [Stages](#stage) in the Arthur platform). Can be categorical or continuous.
+Example:
+> The attribute `age` is an input to the model, whereas the attribute `creditworthy` is the target for the model.
+
+Synonyms: variable, {predictor, input}, {ouput, target}, prediction.
+
+Related terms: [input](#input), [stage](#stage), [prediction](#prediction), [ground truth](#ground-truth)
+
+
+## Bias<a name="bias"></a>
+While bias is an overloaded term in stats&ML, we refer specifically to where a model's outcomes have the potential to differentially harm certain subgroups of a population.
+
+Example:
+> This credit approval model tends to lead to biased outcomes: men are approved for loans at a rate 50% higher than women are.
+
+
+Related terms: [bias detection](#bias-detection-a-name-bias-detection-a), [bias mitigation](#bias-mitigation-a-name-bias-mitigation-a), [disparate impact](#disparate-impact-a-name-disparate-impact-a)
+
+
+## Bias Detection<a name="bias-detection"></a>
+The detection and quantification of {doc}`algorithmic bias </user-guide/walkthroughs/bias_monitoring>` in an ML system, typically as evaluated on a model's outputs (predictions) across different populations of a sensitive attribute.
+[Many definitions](http://fairware.cs.umass.edu/papers/Verma.pdf) of algorithmic bias have been proposed, including group fairness and individual fairness defintions.
+Group fairness definitions are often defined by comparing group-conditional statistics about the model's predictions.
+In the below definitions, the group membership feature is indicated by $G$ and a particular group membership value is indicated by $g$.
+
+Example:
+> Common metrics for group fairness include Demographic Parity, Equalized Odds, and Equality of Opportunity.
+
+Related terms: [bias mitigation](#bias-mitigation)
+
+### Demographic Parity
+
+A fairness metric which compares group-conditional selection rates. The quantity being compared is:
+
+$$
+\begin{align*}
+\mathbb P(\hat Y = 1 | G = g)
+\end{align*}
+$$
+
+There is not necessarily a normative ideal relationship between the selection rates for each group:
+in some situations, such as the allocation of resources, it may be important to minimize the disparity
+in selection rates across groups; in others, metrics based on group-conditional accuracy may be more relevant.
+However, even in the latter case, understanding group-conditional selection rates, especially when
+compared against the original training data, can be useful contextualization for the model and its task as a whole.
+
+Related term: [disparate impact](#disparate-impact-a-name-disparate-impact-a)
+
+### Equal Opportunity
+
+A fairness metric which compares group-conditional true positive rates. The quantity being compared is:
+
+$$
+\begin{align*}
+\mathbb P(\hat Y = 1 | Y = 1, G = g)
+\end{align*}
+$$
+
+For all groups, a true positive rate closer to 1 is better.
+
+### Equalized Odds
+
+A fairness metric which incorporates both group-conditional true positive rates and false positive rates,
+or, equivalently, true positive rates and true negative rates. There are a variety of implementations (due
+to the fact that some quadrants of the confusion matrix are complements of one another); here is one possible quantity to compare across groups:
+
+$$
+\begin{align*}
+\mathbb P (\hat Y = 1 | Y = 1, G = g) + \mathbb P(\hat Y = 0 | Y = 0, G = g)
+\end{align*}
+$$
+
+In this implementation, this quantity should be as close to 2 as possible for all groups.
+
+
+## Bias Mitigation<a name="bias-mitigation"></a>
+Automated techniques to mitigating bias in a discriminatory model. Can be characterized by where the technique sits in the model lifecycle:
+ * **Pre-Processing**: Techniques that analyze datasets and often modify/resample training datasets so that the learned classifier is less discriminatory.
+ * **In-Processing**: Techniques for training a fairness-aware classifier (or regressor) that explicitly trades off optimizing for accuracy and also maintaining fairness across sensitive groups.
+ * **Post-Processing**: Techniques that only adjust the output predictions from a discriminatory classifier, without modifying the training data or the classifier.
+
+Related terms: [bias detection](#bias-detection)
+
+
+
+
+## Binary Classification<a name="binary-classification"></a>
+A modeling task where the target variable belongs to a discrete set with two possible outcomes.
+
+Example:
+> This binary classifier will predict whether or not a person is likely to default on their credit card.
+
+
+Related terms: [output type](#output-type), [classification](#classification), [multilabel classification](#multilabel-classification)
+
+
+
+## Categorical Attribute<a name="categorical-attribute"></a>
+An attribute whose value is taken from a discrete set of possibilities.
+
+Example:
+> A person's blood type is a categorical attribute: it can only be A, B, AB, or O.
+
+Synonyms: discrete attribute
+
+Related terms: [attribute](#attribute), [continuous](#continuous-attribute), [classification](#classification)
+
+## Classification<a name="classification"></a>
+A modeling task where the target variable belongs to a discrete set with a fixed number of possible outcomes.
+
+Example:
+> This classification model will determine whether an [input](#input) image is of a cat, a dog, or fish.
+
+Related terms: [output type](#output-type), [binary classification](#binary-classification), [multilabel classification](#multilabel-classification)
+
+## Continuous Attribute<a name="continuous-attribute"></a>
+An attribute whose value is taken from an ordered continuum, which can be bounded or unbounded.
+
+Example:
+> A person's height, weight, income, IQ can all be through of as continuous attributes.
+
+Synonyms: numeric attribute
+
+Related terms:  [attribute](#attribute), [continuous](#continuous-attribute), [regression](#regression)
+
+(glossary_custom_role)=
+## Custom Role<a name="custom-role"></a>
+Custom Roles are a resource in the Arthur platform used for managing access control policies when SSO authentication has been enabled. For more instructions on configuring Custom Roles, view the {doc}`custom authorization documentation section </platform-management/access-control-overview/sso-access-control/custom_rbac>`.
+
+(glossary_data_drift)=
+## Data Drift
+Refers to the problem arising when, after a trained model is deployed, changes in the external world lead to degradation of model performance and the model becoming stale.  Detecting data drift will provide a leading indicator about data stability and integrity.
+
+Data drift can be quantified with respect to a specific reference set (eg. the model's training data), or more generally over any temporal shifts in a variable with respect to past time windows.
+
+Your project can {doc}`query data drift metrics through the Arthur API </user-guide/api-query-guide/data_drift>`. This section will provide overview of the available data drift metrics in Arthur's query service.
+
+Related terms: [out of distribution](#ood-detection)
+
+### Definitions
+
+#### P and Q<a name="p-and-q"></a>
+
+We establish some mathematical housekeeping for the below metrics. Let $P$ be the reference distribution and $Q$ be the target distribution. These are both probability distributions that can be approximated by binning the underlying reference and target sets. Generally, $P$ is an older dataset and $Q$ is a new dataset of interest. We'd like to quantify how far the distributions differ to see if the reference set has gone stale and algorithms trained on it should not be used to perform inferences on the target dataset.
+
+#### Entropy<a name="entropy"></a>
+
+Let $\text{H}(P)$ be the entropy of distribution $P$. It is interpreted as the expected (i.e. average) number of bits (if log base 2) or nats (if log base $e$) required to encode information of a datapoint from distribution $P$. Arthur applications use log base $e$, so interpretation will be in nats.
+
+$$
+\begin{align*}
+\text{H}(P)
+= -\sum_{k=1}^K P(x_k)*\text{log}P(x_k)
+= -\text{E}_P[\text{log}P(x_k)]
+\end{align*}
+$$
+
+### KL Divergence<a name="kl-divergence"></a>
+
+Let $\text{D}(P \parallel Q)$ be the Kullback-Leibler (KL) Divergence from $P$ to $Q$. It is interpreted as the nats of information we expect to lose in using $Q$ instead of $P$ for modeling data $X$, discretized over probability space $K$. KL Divergence is not symmetrical, i.e. $\text{D}(P \parallel Q) \neq \text{D}(Q \parallel P)$, and should not be used as a distance metric.
+
+$$
+\begin{align*}
+\text{D}(P||Q)
+= \sum_{k=1}^K P(x_k)*(\text{log}P(x_k)-\text{log}Q(x_k)) \\
+= \text{E}_P[\text{log}P(x)-\text{log}Q(x)]
+\end{align*}
+$$
+
+### Population Stability Index (PSI)<a name="psi"></a>
+
+Let $\text{PSI}(P,Q)$ be the Population Stability Index (PSI) between $P$ and $Q$. It is interpreted as the roundtrip loss of nats of information we expect to lose from $P$ to $Q$ and then from $Q$ returning back to $P$, and vice versa. PSI smooths out KL Divergence since the return trip information loss is included, and this metric is popular in financial applications.
+
+$$
+\begin{align*}\text{PSI}(P,Q)
+= \text{D}(P||Q) + \text{D}(Q||P) \\
+= \sum_{k=1}^K (P(x_k)-Q(x_k))*(\text{log}P(x_k)-\text{log}Q(x_k)) \\
+= \text{E}_P[\text{log}P(x)-\text{log}Q(x)]+\text{E}_Q[\text{log}Q(x)-\text{log}P(x)]
+\end{align*}
+$$
+
+### JS Divergence<a name="js-divergence"></a>
+
+Let $\text{JSD}(P,Q)$ be the Jensen-Shannon (JS) Divergence between $P$ and $Q$. It smooths out KL divergence using a mixture of the base and target distributions and is interpreted as the entropy of the mixture $M=\frac{P+Q}{2}$ minus the mixture of the entropies of the individual distributions.
+
+$$
+\begin{align*}\text{JSD}(P,Q)
+= \frac{1}{2}\text{D}(P||M) +  \frac{1}{2}\text{D}(Q||M) \\
+= \text{H}(\frac{P+Q}{2})-\frac{\text{H}(P)+H(Q)}{2}
+\end{align*}
+$$
+
+### Hellinger Distance<a name="hellinger-distance"></a>
+
+Let $\text{HE}(P,Q)$ be the Hellinger Distance between $P$ and $Q$. It is interpreted as the Euclidean norm of the difference of the square root distributions of $P$ and $Q$.
+
+$$
+\begin{align*}
+\text{HE}(P,Q)
+= {\frac {1}{\sqrt {2}}}{\bigl \|}{\sqrt {P}}-{\sqrt {Q}}{\bigr \|}_{2} \\
+= {\frac {1}{\sqrt {2}}}{\sqrt {\sum _{k=1}^{K}\left({\sqrt {P(x_k)}}-{\sqrt {Q(x_k)}}\right)^{2}}}
+\end{align*}
+$$
+
+(glossary_hypothesis_test)=
+### Hypothesis Test
+
+Hypothesis testing uses different tests depending on whether a feature is categorical or continuous.
+
+For categorical features, let $\chi_{\text{K}-1}^2(P,Q)$ be the chi-squared test statistic for $P$ and $Q$, with $\text{K}$ being the number of categories of the feature, i.e. $\text{K}-1$ are the degrees of freedom. Let $\text{N}_{Pk}$ and $\text{N}_{Qk}$ be the count of occurrences of feature being $k$, with $1\leq k \leq K$, for $P$ and $Q$ respectively. The chi-squared test statistic is the summation of the standardized differences of expected counts between $P$ and $Q$.
+
+$$
+\begin{align*}
+\chi_{K-1}^2(P,Q)
+= \sum_{k=1}^K \frac{(\text{N}_{Qk}-\text{N}_{Pk})^2}{\text{N}_{Pk}}\\
+\end{align*}
+$$
+
+For continuous features, let $\text{KS}(P, Q)$ be the Kolmogorov-Smirnov test statistic for $P$ and $Q$. Let $F_P$ and $F_Q$ be the empirical cumulative density, for $P$ and $Q$ respectively. The Kolmogorov-Smirnov test is a nonparametric, i.e. distribution-free, test that compares the empirical cumulative density functions of $P$ and $Q$.
+
+$$
+\begin{align*}
+\text{KS}(P,Q)
+= \sup_x (F_P(x) - F_Q(x))  \\
+\end{align*}
+$$
+
+The returned test statistic is then compared to cutoffs for significance. A higher test statistic indicates more data drift. We've abstracted the calculations away for you in our query endpoint.
+
+For `HypothesisTest`, the returned value is transformed as -log_10(P_value) to maintain directional parity with the other data drift metrics. That is, lower P_value is more significant and implies data drift, reflected in a higher -log_10(P_value).
+
+### Multivariate<a name="multivariate"></a>
+
+Arthur also offers a multivariate Anomaly Score which you can configure via the steps detailed {ref}`here <enrichments_anomaly_detection>`.
+See {ref}`here <arthur_algorithms_anomaly_detection>` for an explanation of how these scores are calculated.
+
+## Disparate Impact<a name="disparate-impact"></a>
+Legal terminology orginally from Fair Lending case law. This constraint is strictly harder than [Disparate Treatment](#disparate-treatment) and asserts that model outcomes must not be discriminatory across protected groups. That is, the outcome of a decisioning process should not be substantially higher (or lower) for one group of a protected class over another.
+
+While there does not exist a single threshold for establishing the presence or absence of disparate impact, the so-called ["80% rule"](https://www.eeoc.gov/laws/guidance/questions-and-answers-clarify-and-provide-common-interpretation-uniform-guidelines) is commonly referenced. However, we strongly recommend against adopting this rule-of-thumb, as these analyses should be grounded in use-case specific analysis and the legal framework pertinent to a given industry.
+
+Example:
+> Even though the model didn't take `gender` as input, it still results in disparate impact when we compare outcomes for males and females.
+
+Related terms: [bias](#bias), [disparate treatment](#disparate-treatment)
+
+
+## Disparate Treatment<a name="disparate-treatment"></a>
+Legal terminology originally from Fair Lending case law. Disparate Treatment asserts that you are not allowed to consider protected variables (eg race, age, gender) when approving or denying an applicant for a credit card loan.
+In practical terms, this means that a data scientist cannot include these attributes as [inputs](#input) to a credit decisioning model.
+
+Adherence to Disparate Treatment is not a sufficient condition for actually acheiving a fair model (see [proxy](#proxy) and [bias detection](#bias-detection)). "Fairness through unawareness" is not good enough.
+
+Related terms: [bias](#bias), [disparate impact](#disparate-impact)
+
+## Enrichment<a name="enrichment"></a>
+Generally used to describe data or metrics added to raw data after ingestion. Arthur provides various enrichments such as Anomaly Detection and Explainability. See {doc}`/user-guide/walkthroughs/enrichments` for details around using enrichments within Arthur.
+
+## Feature<a name="feature"></a>
+An individual attribute that is an input to a [model](#arthur-model)
+
+Example:
+> The credit scoring model has features like `“home_value”`, `“zip_code”`, `“height"`.
+
+## Ground Truth<a name="ground-truth"></a>
+The true label or target-variable (Y) corresponding to inputs (X) for a dataset.
+
+
+Examples:
+  ```
+pred = sklearn_model.predict_proba(X)
+arthur_model.send_inference(
+    model_pipeline_input=X,
+    predicted_values={1:pred, 0: 1-pred})
+```
+
+Related terms: [prediction](#prediction)
+
+
+## Image Data<a name="image-data"></a>
+Imagery data commonly used for computer vision models.
+
+Related terms: [attribute](#attribute), [output type](#output-type), [Stage](#stage)
+
+
+
+## Inference<a name="inference-data"></a>
+One row of a dataset. An inference refers to passing a single input into a model and computing the model's prediction. Data associated with that inference might include (1) input data, (2) model's prediction, (3) corresponding ground truth.
+With respect to the Arthur platform, the term inference denotes any and all of those related components of data for a single input&prediction.
+
+
+Related terms: [ArthurInference](#arthur-inference), [stage](#stage)
+
+
+
+## Input<a name="input"></a>
+A single instance of data, upon which a model can calculate an output [prediction](#prediction). The input consists of all relevant [features](#feature) together.
+
+Example:
+> The input features for the credit scoring model consist of `“home_value”`, `“zip_code”`, `“height"`.
+
+Related terms: [feature](#feature), [model](#arthur-model)
+
+## Input Type<a name="input-type"></a>
+For an [ArthurModel](#arthur-model), this field declares what kind of input datatype will be flowing into the system.
+
+Allowable values are defined in the `InputType` enum:
+* [`Tabular`](#tabular-data)
+* [`Image`](#image-data)
+* [`NLP`](#nlp-data)
+
+Example:
+```python
+arthur_model = connection.model(name="New_Model",
+                               input_type=InputType.Tabular,
+                               model_type=OutputType.Regression)
+```
+
+Related terms: [output type](#output-type), [tabular data](#tabular-data), [nlp data](#nlp-data)
+
+
+
+## Model Health Score<a name="model-health-score"></a>
+On the UI dashboard, you will see a model health score between 0-100 for each of your models.
+The score is an aggregation based on the following metrics: performance, drift and ingestion.
+Each metric for a model is computed every hour, combined, and then aggregated by computing the average over a 30-day window.
+The thresholds for model-health are (0-32, 33-65, 66-100) for (Red, Yellow, Green) respectively.
+
+* Performance:
+  * Regression: 1 - Normalized MAE
+  * Classification: F1 Score
+
+* Drift
+  * 1 - Average Anomaly Score
+
+* Ingestion
+  * Variance of normalized time periods between ingestion events
+  * Variance of normalized volume differences between ingestion events
+
+
+You can extract the health score via an API [call](../api-documentation/v3-api-docs.html#tag/models/paths/~1models~1health/get) as well.
+
+
+## Model Onboarding<a name="model-onboarding"></a>
+Model onboarding refers to the process of defining an `ArthurModel`, preparing it with the necessary reference dataset, passing it through a validation check, and saving it to the Arthur system.
+
+Once your model is onboarded onto Arthur, you can use the Arthur system to track the model and view all its performance and analytics in your online Arthur dashboard.
+
+Related terms: [ArthurModel](#arthur-model), [reference dataset](#reference-data)
+
+
+## Multiclass Classification<a name="multiclass-classification"></a>
+A modeling task where each [input](#input) is associated with one label, from a fixed set of possible labels.
+Often this is a binary classifier (the output is either 0 or 1), but the output can also have more than 2 possible labels.
+
+Example:
+> This NLP model applies the most relevant tag to news articles. The model is trained on example articles which are tagged with a topic like `Congress`.
+
+Related terms: [multilabel clasification](#multilabel-classification), [output type](#output-type),
+
+
+## Multilabel Classification<a name="multilabel-classification"></a>
+A modeling task where each [input](#input) is associated with two or more labels, from a fixed set of possible labels.
+
+Example:
+> This NLP model applies relevant tags to news articles. The model is trained on example articles which are tagged with multiple topics like `Politics, Elections, Congress`.
+
+Related terms: [output type](#output-type), [multiclass clasification](#multiclass-classification)
+
+
+## NLP Data<a name="nlp-data"></a>
+Unstructured text sequences commonly used for Natural Language Processing models.
+
+Related terms: [attribute](#attribute), [output type](#output-type), [Stage](#stage)
+
+
+## Non-Input Attribute <a name="non-input-attribute"></a>
+
+A non-input attribute is an attribute that an ArthurModel will track that does not actually enter the model as an input.
+
+Common non-input attributes are protected class attributes such as age, race, or sex. By sending such non-input attributes to Arthur, you can track model performance based on these groups in your data to evaluate model bias and fairness.
+
+Related terms: [attribute](#attributea-nameattributea), [bias](#biasa-namebiasa)
+
+
+## Object Detection<a name="object-detection"></a>
+The OutputType for computer vision models with the purpose of detecting an object within an image and outputting a box which bounds the object.
+
+This bounding box is used to identify _where_ the object resides in the image.
+
+Related terms: [image](#image-dataa-nameimage-dataa)
+
+(glossary_organization)=
+## Organization<a name="organization"></a>
+An Organization is a structural entity utilized by the Arthur platform to organize and manage access to resources that exist on the platform. Users are added to Arthur Organizations with a given role that provides them with Read and/or Write access to some subset of that Organization's resources, as defined by the User's role.
+
+Each Organization has its own:
+- [Models](#arthur-modela-namearthur-modela)
+- [Model Groups](#arthur-model-groupa-namearthur-model-groupa)
+- [Service Accounts](#service-accounta-nameservice-accounta)
+- {ref}`Custom Roles <glossary_custom_role>`
+
+## Out of Distribution Detection<a name="ood-detection"></a>
+Refers to the challenge of detecting when an input (or set of inputs) is substantially different from the distribution of a larger set of reference inferences. This term commonly arises in the context of data drift, where we want to detect if new inputs are different from the training data (and distribution thereof) for a particular model. OOD Detection is a relevant challenge for Tabular data as well as unstructured data such as images and sequences.
+
+
+Related terms: {ref}`glossary_data_drift`
+
+## Output Type<a name="output-type"></a>
+For an [ArthurModel](#arthur-model), this field declares what kind of output predictions will be flowing out of the system.
+
+Allowable values are defined in the `OutputType` enum:
+  * [`Regression`](#regression)
+    * appropriate for continuous-valued targets
+  * [`Multiclass`](#classification)
+    * appropriate for both binary classiers and multiclass classifiers
+  * [`Multilabel`](#multilabel-classification)
+    * appropriate for multilabel classifiers
+  * [`ObjectDetection`](#object-detection)
+    * only available for computer vision models
+
+Example:
+```python
+arthur_model = connection.model(name="New_Model",
+                               input_type=InputType.Tabular,
+                               output_type=OutputType.Regression)
+```
+
+
+Related terms: [input type](#input-type)
+
+## Prediction<a name="prediction"></a>
+The output prediction (y_hat) of a trained model for any input.
+
+Related terms: [ground truth](#ground-truth)
+
+## Protected Attribute<a name="protected-attribute"></a>
+An attribute of an inference that is considered sensitive with respect to model bias. Common examples include race, age, and gender. The term "protected" comes from the Civil Rights Act of 1964.
+
+Synonyms: sensitive attribute
+
+Related terms: [bias](#bias), [proxy](#proxy)
+
+
+## Proxy<a name="proxy"></a>
+An input attribute in a model (or combination thereof) that is highly correlated with a protected attribute such as race, age, or gender. The presence of proxies in a dataset makes it difficult to rely only on [Disparate Treatment] as a standard for fair ML.
+
+Example:
+> In most US cities, zip code is a strong proxy for race. Therefore, one must be cautious when using zip code as an input to a model.
+
+Related terms: [bias](#bias), [disparate impact](#disparate-impact), [disparate treatment](#disparate-treatment)
+
+
+## Reference <a name="reference-data"></a>
+The dataset used as a baseline reference for an ArthurModel.
+
+A reference dataset must include a sample of the input features a model receives.
+
+A reference dataset can optionally include a sample of model outputs, ground truth values, and other non-input attributes as metadata.
+
+The reference dataset for a model is used to compute drift: the distribution of input features in the reference dataset makes up the baseline against which future inferences are compared to compute anomaly scores.
+
+Related terms: [inference](#inference)
+
+## Regression<a name="regression"></a>
+A modeling task (or model) where the [target variable](#ground-truth) is a continuous variable.
+
+Example:
+> This regression model predicts what the stock price of $AAPL will be tomorrow.
+
+Related terms: [output type](#output-type)
+
+## Sensitive Attribute<a name="sensitive-attribute"></a>
+See [protected attribute](#protected-attribute)
+
+## Service Account<a name="service-account"></a>
+Service Accounts are entities within the Arthur Platform that provide access to Arthur APIs for automated systems (machine-to-machine communication). They represent both an identity as well as a set of permissions for that identity. Each Service Account has an access token that includes a specific role, which grants access to the Arthur APIs. The roles used in Service Accounts provide access within a single Arthur {ref}`Organization <glossary_organization>`.
+
+## Stage<a name="stage"></a>
+Taxonomy used by the Arthur platform to delineate how [attributes](#attribute) contribute to the model computations.
+Allowable values are defined in the `Stage` enum:
+  * `ModelPipelineInput` : Input to the entire model pipeline. This will most commonly be the Stage used to represent all model inputs. Will contain base input features that are familiar to the data scientist: categorical and continuous columns of a tabular dataset.
+  * `PredictedValue`: The predictions coming out of the model.
+  * `GroundTruth`: The ground truth (or target) attribute for a model. Must be one-hot for classifiers
+  * `GroundTruthClass`: The ground truth class for classification models, not one-hot encoded
+  * `NonInput`: Ancillary data that can be associated with each inference, but not necessarily a direct input to the model. For example, sensitive attributes like `age`, `sex`, or `race` might not be direct model inputs, but will useful to associate with each prediction.
+
+## Tabular Data<a name="tabular-data"></a>
+Data type for model inputs where the data can be thought of as a table (or spreadsheet) composed of rows and columns. Each column represents an input [attribute](#attribute) for the model and each row represents a separate record that composes the training data. In supervised learning, exactly one of the columns acts as the [target](#ground-truth).
+
+Example:
+> This credit scoring model is trained on tabular data. The [input](#input) attributes are `income`, `country`, and `age` and the [target](#ground-truth) is FICO score.
+
+
+Related terms: [Attribute](#attribute), [output type](#output-type), [Stage](#stage)
+
+## Version Label<a name="version-label"></a>
+
+A Version Label is a string that can represent a custom version for your Arthur Model within its Arthur Model Group. Version Labels are not required and the platform will default to using the Version Sequence Number when not provided.
+
+Example:
+
+```python
+# retrieve the model group
+model_group = connection.get_model_group(model_group_id)
+
+# create the new version of the model
+arthur_model_v2 = connection.model(name="Model_V2",
+                                   input_type=InputType.Tabular,
+                                   model_type=OutputType.Regression)
+
+# add the new model to the model group
+model_group.add_version(arthur_model_v2, label="2.0.1")
+label = arthur_model_v2.version_label
+arthur_model_v2.save()
+# label == "2.0.1"
+```
+
+Related terms: [Arthur Model](#arthur-model), [Arthur Model Group](#arthur-model-group), [Version Sequence Number](#version-sequence-number)
+
+## Version Sequence Number<a name="version-sequence-number"></a>
+
+A Version Sequence Number is a unique, auto-incrementing (starting at 1) integer that is assigned to Arthur Models in an Arthur Model Group. This number uniquely represents an Arthur Model’s Version within the Model Group. In the case a Version Label is not provided, the platform will show the Version Sequence Number instead.
+
+Example:
+
+```python
+# retrieve the first version of a model
+arthur_model_v1 = connection.get(model_id)
+num = arthur_model_v1.version_sequence_num
+# num == 1
+
+# retrieve the second version of a model
+model_group = arthur_model_v1.model_group
+arthur_model_v2 = model_group.get_version(sequence_num=2)
+num = arthur_model_v2.version_sequence_num
+# num == 2
+```
+
+Related terms: [Arthur Model](#arthur-model), [Arthur Model Group](#arthur-model-group), [Version Label](#version-label)

files/arthur-docs-markdown/more-info/index.md.txt

@@ -0,0 +1,26 @@
+# More Information
+
+## {doc}`algorithms`
+
+The {doc}`algorithms` page describes the math behind the algorithms Arthur implements for 
+calculating performance-related metrics and insights for your models.
+
+## {doc}`FAQs`
+
+Visit our {doc}`FAQs` page to see answers to common questions about Arthur and model 
+monitoring.
+
+## {doc}`glossary`
+
+Our {doc}`glossary` contains a definition of terms used throughout the Arthur documentation, as well as 
+related links to other terms and guides to get familiar with Arthur.
+
+
+```{toctree}
+:maxdepth: 2
+:hidden:
+
+Arthur Algorithms <algorithms>
+FAQs <FAQs>
+Glossary <glossary>
+```

files/arthur-docs-markdown/not-found.rst.txt

@@ -0,0 +1,6 @@
+:orphan:
+
+Not Found
+=========
+
+The page you're looking for wasn't found.

files/arthur-docs-markdown/platform-management/access-control-overview/index.md.txt

@@ -0,0 +1,43 @@
+# Access Control Overview
+
+Arthur has supports a variety of mechanisms for authentication, who a user is, and authorization, what a user can do
+(RBAC).
+By default, Arthur will use a built-in authentication and authorization system. In on-prem installations, cluster
+administrators can optionally configure an external Identity Provider (IdP) to control user authentication and
+authorization. See the sections below for an overview of Arthur Standard Access Control and Arthur SSO Access Control.
+
+## {doc}`Arthur Standard Access Control <standard_access_control>`
+
+The {doc}`Arthur Standard Access Control <standard_access_control>` page shows how to use the default access control
+system in Arthur.
+
+## {doc}`Single-Sign-On (SSO) Access Control <sso-access-control/index>` (On-prem only)
+
+The {doc}`Single-Sign-On (SSO) Access Control <sso-access-control/index>` page shows how to set up and use an external
+IdP for access
+control.
+
+## Access Control Paradigm Comparison
+
+The following table shows a high level comparison of the different capabilities in
+{doc}`Arthur Standard Access Control <standard_access_control>`
+and {doc}`Arthur SSO Access Control <sso-access-control/index>`.
+
+| Capability                                             | Arthur Standard Access Control | SSO Access Control         |
+|--------------------------------------------------------|--------------------------------|----------------------------|
+| User creation                                          | In the Arthur UI               | In the IdP                 |
+| User sign-in                                           | In the Arthur UI               | In the IdP                 |
+| User password reset                                    | In the Arthur UI               | In the IdP                 |
+| Can use Arthur API Keys                                | Yes                            | No                         |
+| Supports Custom Roles and RBAC                         | No                             | Yes                        |
+| Users can have access to multiple Arthur organizations | Yes                            | Yes                        |
+| Availability                                           | All Arthur installations       | On-prem only               |
+| Required permissions to configure                      | Organization Admin             | Installation/Cluster Admin |
+
+```{toctree}
+:maxdepth: 2
+:hidden:
+
+Default Access Control <standard_access_control>
+SSO Access Control <sso-access-control/index>
+```

files/arthur-docs-markdown/platform-management/access-control-overview/sso-access-control/custom_rbac.md.txt

@@ -0,0 +1,133 @@
+# Custom RBAC
+
+(custom_rbac_managing_orgs_and_roles)=
+## Managing RBAC and Organizations for SSO users
+
+For customers using SSO (on-prem only), Arthur provides the ability to set up a fully customizable RBAC.
+Please follow the below:
+
+1. When setting up your identity provider via the YAML configuration, supply a global
+   role name and set of permissions under `globalRoleDefs` that your identity provider will authenticate users with.
+   This configuration will create the global role in the Arthur authorization system when it is applied. See the
+   {ref}`Creating Global Roles for Managing Organizations and RBAC Policies Guide <creating_global_roles_in_arthur_config>` for
+   more information.
+2. That global role can then create custom role mappings for each organization:
+    * During organization creation, include the role configuration JSON (see below for example) in the request body when
+      calling [`POST /organizations`](../../../../api-documentation/v3-api-docs.html#tag/organizations/paths/~1organizations/post).
+    * After an organization is created, create or add custom_roles by sending the role configuration JSON (see below for
+      example) in the request body when
+      calling [`POST /autorization/custom_roles`](../../../../api-documentation/v3-api-docs.html#tag/authorization/paths/~1authorization~1custom_roles/post).
+3. Users logging in through your IdP must now have a valid known role in their token when accessing the Arthur
+   Platform. Arthur will use this role to both authenticate that the user is a member of the organization and to
+   determine the permissions they have.
+
+## Managing Roles and Permissions
+
+### Understanding Permissions
+
+* A permission is a combination of a resource and an action. For
+  example `raw_data read`, `users write`, `models delete`.
+* For a full list of available permissions. please see {doc}`here </platform-management/reference/permissions_set>`.
+* For a directory of permissions by API endpoint, please see
+  {doc}`here </platform-management/reference/permissions_by_endpoint>`.
+
+### Create Custom Roles
+
+The [`POST /autorization/custom_roles`](../../../../api-documentation/v3-api-docs.html#tag/authorization/paths/~1authorization~1custom_roles/post)
+endpoint is available for customers using SSO to operate on custom roles for each organization. A few notes:
+
+* This endpoint only operates on permission scopes within each organization. Permissions that have global scope (such as
+  creating a new organization) cannot be granted via this endpoint, those permissions must be assigned to a role with
+  global privileges via the Arthur IdP configuration YAML. See
+  {ref}`Creating Global Roles for Managing Organizations and RBAC Policies Guide <creating_global_roles_in_arthur_config>` for more
+  information.
+* Roles can have a list of permissions to allow and/or a list of other roles to inherit permissions from.
+* Role names cannot conflict with {doc}`default roles </platform-management/reference/permissions_by_standard_roles>`.
+* Supplied permissions must be valid known Arthur permissions.
+* Roles can inherit the permissions of other roles that are either
+  {doc}`default roles </platform-management/reference/permissions_by_standard_roles>`, or roles also defined in the same
+  organization. Unknown inherited role names will be rejected.
+
+### Get Custom Roles
+
+To retrieve a list of roles defined for an organization,
+use: [`GET /autorization/custom_roles`](../../../../api-documentation/v3-api-docs.html#tag/authorization/paths/~1authorization~1custom_roles/get).
+To filter on specific roles pass a comma separated list of role names in a roles query parameter. For
+example:`/authorization/custom_roles?roles=role1,role2`. If you wish to return all roles simply leave out the query
+parameter or pass `"*"` as role.
+
+### Delete Custom Roles
+
+To delete a role or multiple roles from an organization,
+use [`DELETE /autorization/custom_roles`](../../../../api-documentation/v3-api-docs.html#tag/authorization/paths/~1authorization~1custom_roles/delete).
+Specify which roles to delete in the JSON request body. For example, to delete a single role:
+
+```json
+{
+  "roles": [
+    "role3"
+  ]
+}
+```
+
+To delete all roles pass "*". 
+
+```{warning} If you do not specify an organization_id, this will delete all custom roles you have created
+``` 
+
+```json
+{
+  "roles": [
+    "*"
+  ]
+}
+```
+
+### Example Role Configuration JSON
+
+Below is an example JSON request body that creates three roles. role1 has 3 permissions defined, role2 gets an
+additional permission and then inherits the 3 permissions from role1, and role3 inherits the permissions from Arthur's
+default "Model Owner" role. For more details on the expected schema for each endpoint,
+see [Authorization API documenation](../../../../api-documentation/v3-api-docs.html#tag/authorization).
+
+```json
+{
+  "roles": [
+    {
+      "role_name": "role1",
+      "permissions": [
+        {
+          "resource": "metric_data",
+          "action": "read"
+        },
+        {
+          "resource": "metric_data",
+          "action": "write"
+        },
+        {
+          "resource": "tag",
+          "action": "read"
+        }
+      ]
+    },
+    {
+      "role_name": "role2",
+      "permissions": [
+        {
+          "resource": "user_self",
+          "action": "read"
+        }
+      ],
+      "inherited_role_names": [
+        "role1"
+      ]
+    },
+    {
+      "role_name": "role3",
+      "inherited_role_names": [
+        "Model Owner"
+      ]
+    }
+  ]
+}
+```

files/arthur-docs-markdown/platform-management/access-control-overview/sso-access-control/index.md.txt

@@ -0,0 +1,322 @@
+# Arthur SSO Access Control Overview
+
+In on-prem installations, cluster administrators have the ability to configure their Arthur install to use a 3rd party
+identity provider (IdP) to manage Arthur users and access. Arthur can work with any IdP that supports the Open ID
+Connect (OIDC) or SAML protocols.
+
+## User and App Authentication
+
+Regardless of if your IdP is using OIDC or SAML, the IdP will be responsible for authenticating users and system access
+to Arthur. As a result, applications or downstream systems that need to authenticate with Arthur will require working
+with your IdP to get access. Some IdPs provide support for AppID's or service accounts, but it will depend on the
+specific IdP and how your organization supports it.
+
+```{warning}
+Important! Arthur API keys cannot be used when SSO in enabled. 
+Instead, users will rely on their own authentication tokens, which are validated by the IdP and enforce the custom 
+permissions assigned to their roles.
+```
+
+## Authorization (RBAC)
+
+Arthur installations with SSO are able to use custom role-based access control policies (RBAC) for
+users authenticated via the IdP. Arthur's custom RBAC system uses two kinds of roles. The first kind are organization
+level roles, which grant a user the specified permissions within a single Arthur organization. The second kind are
+global roles, which grant the specified privileges across all organizations in the system. Global roles are intended for
+administrators, so they can perform actions like managing organizations and roles at the global level. They should not
+be used to grant non-admin users cross-organization access. Instead, users that need access to more than one
+organization should have a distinct role for each organization. Commonly, the global administrator role only has
+permissions to create new organizations and to manage custom roles for the platform. All model and data access should
+be configured using organization scoped roles. See the {doc}`custom_rbac` for an overview of how to configure
+custom roles for your Arthur installation.
+
+### Linking IdP Groups to Arthur Roles
+
+Roles are the link between the IdP's user groups and the Arthur access control system. As shown
+in [Set Up an Identity Provider](#set-up-an-identity-provider), when you
+configure an IdP, you will tell Arthur how to parse a user's groups out of the IdP response. The group, or list of
+groups, returned from your IdP must match the names of custom roles configured in Arthur. For this reason, Arthur
+enforces all custom roles to have unique names, so the platform can map a user group from an
+external IdP to a specific set of permissions. This means **all roles in the system must have unique names, even across
+organizations.**
+
+For example, if using OIDC and the IdP returns a JWT token of the following format, Arthur must be configured with roles
+that match the groups listed under the "groups" field in the token. The group names are used to look up the roles that
+apply for the user in Arthur. More examples of how to configure the OIDC JWT claim parsing can be found in
+{doc}`setting_up_oidc_idp`.
+
+```json
+{
+  "sub": "1234567890",
+  "name": "John Doe",
+  "iat": 1516239022,
+  // this field in the token corresponds to the user's groups in the IdP
+  "groups": [
+    // an "idp-admin" role must be created in Arthur to match this group
+    "idp-admin",
+    // an "org-1-user" role must be created in Arthur to match this group
+    "org-1-user"
+  ]
+}
+```
+
+```{note}
+It is ok if the OIDC token contains extra groups that are not matched to roles in Arthur. 
+It is not required that every group have a corresponding role, but at least one group must.
+```
+
+Similarly if using SAML, the SAML assertion from the IdP must contain some attribute values that contains the user's
+groups in the IdP. More examples of how to configure the SAML attribute parsing can be found in
+{doc}`Setting Up a SAML IdP <setting_up_saml_idp>`.
+
+```xml
+
+<saml2:AttributeStatement>
+    <saml2:Attribute Name="email">
+        <saml2:AttributeValue>support@arthur.ai</saml2:AttributeValue>
+    </saml2:Attribute>
+    <saml2:Attribute Name="firstName">
+        <saml2:AttributeValue>John</saml2:AttributeValue>
+    </saml2:Attribute>
+    <saml2:Attribute Name="lastName">
+        <saml2:AttributeValue>Doe</saml2:AttributeValue>
+    </saml2:Attribute>
+    // this field in the assertion corresponds to the user's groups in the IdP
+    <saml2:Attribute Name="groups">
+        // an "idp-admin" role must be created in Arthur to match this group
+        <saml2:AttributeValue>idp-admin</saml2:AttributeValue>
+        // an "org-1-user" role must be created in Arthur to match this group
+        <saml2:AttributeValue>org-1-user</saml2:AttributeValue>
+    </saml2:Attribute>
+</saml2:AttributeStatement>
+```
+
+```{note}
+It is ok if the SAML assertion contains extra groups that are not matched to roles in Arthur. 
+It is not required that every group have a corresponding role, but at least one group must.
+```
+
+## Set Up an Identity Provider
+
+### {doc}`OIDC <setting_up_oidc_idp>`
+
+See {doc}`Setting Up an OIDC IdP <setting_up_oidc_idp>` for the steps to configure an OIDC IdP with Arthur.
+
+### {doc}`SAML <setting_up_saml_idp>`
+
+See {doc}`Setting Up a SAML IdP <setting_up_saml_idp>` for the steps to configure a SAML IdP with Arthur.
+
+## Creating Custom Roles
+
+As mentioned above, custom roles can be used to grant either permissions within an organization or permissions across
+all organizations. This section will first cover how to create global roles, then it will cover how to create roles for
+an organization. The steps outlined here are sufficient to get SSO working. For more details about managing custom RBAC
+access control, please see the {doc}`custom_rbac` guide.
+
+(creating_global_roles_in_arthur_config)=
+
+### Creating Global Roles for Managing Organizations and RBAC Policies
+
+```{note}
+In order to complete this section, you must have access to the Arthur Admin Console config page.
+```
+
+Global roles are intended to be used by a cluster administrator to manage organizations and roles in the Arthur
+application. Global roles should be scoped with the minimal privileges and should be given out with caution.
+**It is not recommended to grant model or data access with global roles as that would give access to every model in the
+system.** Global roles can be created in the Arthur installer Admin Console configuration in the "Use a 3rd Party
+Global Identity Provider" section where the IdP configuration YAML is set. A list of custom global roles can be provided
+in the Identity Provider YAML under the `globalRoleDefs` key. The following
+example shows a global role for an Arthur cluster admin to manage organizations and RBAC for the
+installation:
+
+```yaml
+  globalRoleDefs:
+    # Here we specify a list to define multiple global roles
+    - name: "idp-admin" # change the name of this role to match the cluster administrator group name in your IdP
+      # this role grants a user permissions to manage organizations and RBAC policies in Arthur 
+      permissions:
+        custom_roles:
+          - read
+          - write
+          - delete
+        organization_global:
+          - read
+          - write
+        organization:
+          - read
+          - delete
+```
+
+Once you add a role to your config, be sure to click "Save" at the bottom of the page and then deploy the change. After
+the update completes, the new global role should be created.
+
+(creating_organization_roles)=
+
+### Creating Organization Scoped Roles for Non-admin Users
+
+Completing this section will require either access to the Arthur superadmin user, or access to a previously created,
+custom global role assumable via an IdP
+(see [above](#creating-global-roles-for-managing-organizations-and-rbac-policies)).
+Additionally, only installations with SSO configured can use custom RBAC policies. If you do not have SSO configured,
+please see {doc}`../standard_access_control`.
+
+The section below assumes an organization already exists that you would like to give users access to. If you would like
+to create a new Arthur organization, please check out the
+{ref}`Managing RBAC and Organizations for SSO users <custom_rbac_managing_orgs_and_roles>`,
+then come back here to add roles to that new organization.
+
+Using either the Arthur superadmin user or having assumed a custom RBAC role with global scope to manage RBAC,
+organization scoped roles can be created using the Arthur API. The following is an example of a request to create a new
+role in an organization. Be sure to fill in the missing values for the hostname,
+authorization token (must be a superadmin or your token from a custom global role), organization ID, and change the role
+names to match your IdP user groups. See
+{ref}`Managing RBAC and Organizations for SSO users <custom_rbac_managing_orgs_and_roles>` for an overview of role
+permissions and some example roles.
+
+```shell
+curl --location 'https://<YOUR ARTHUR HOST HERE>/api/v3/authorization/custom_roles' \
+--header 'Arthur-Organization-ID: <YOUR ORGANIZATION ID HERE>' \
+--header 'Content-Type: application/json' \
+--header 'Authorization: Bearer <YOUR AUTH TOKEN HERE>' \
+--data '{
+    "roles": [
+        {
+            "role_name": "my-idp-group-1",
+            "inherited_role_names": [
+                "Administrator"
+            ]
+        },
+        {
+            "role_name": "my-idp-group-2",
+            "inherited_role_names": [
+                "Model Owner"
+            ]
+        }
+    ]
+}'
+```
+
+This sample API call will create two custom roles in the `<YOUR ORGANIZATION ID HERE>` organization. These
+roles will apply to users in the "my-idp-group-1" or "my-idp-group-2" groups managed by the IdP.
+
+### Granting Users Access to Multiple Organizations
+
+Users can be given access to multiple organizations by creating a unique custom role in each organization, then adding
+them to the associated groups in the IdP. For example, if you have two organizations, `org-1` and `org-2`, the Arthur
+cluster administrator will need to create a custom role in each organization. Let's call them `role-1` and `role-2`
+respectively. As described above, these role names will need to correspond to user groups in the IdP so please name them
+according to your group names. With two organizations and a role created in each, the last step is to add your users to
+both groups in your IdP, and they will be able to access both organizations.
+
+```{note}
+If users have access to more than one organization, they will need to specify one to use during API calls. 
+See the next section for more information on how to specify an organization when using the Arthur API.
+```
+
+## API Authentication with Multiple Organization Users
+
+The Arthur platform separates resources logically by Arthur {ref}`Organizations <glossary_organization>`. When an Arthur
+deployment is configured with Arthur's Standard Authentication mechanism, user access is granted to one organization at
+a time, so all interactions with the platform are automatically scoped with that organization.
+
+However, when SSO is enabled, SSO tokens can grant access to one or more Arthur organizations at a time. They do this by
+listing multiple roles for a user within the token, each with access to a distinct organization. As a result, users can
+have access to more than one organization at a time, and Arthur cannot determine which organization the user intends to
+take an action in. Therefore, when SSO users have access to more than one organization, they must select a specific
+Arthur Organization in each request to the platform via the methods described in this section.
+
+### Alternatives for Selecting an Organization in the Arthur API
+
+1. Request Header
+2. Session-Based Organization Tracking
+3. Arthur SDK Client
+4. Deep Linking to an Organization in the UI
+
+### Request Header
+
+In an API call made by an SSO-authenticated user with access to more than one organization, the platform will first
+check for a request header with an `Arthur-Organization-ID` key. If one is found, the platform will parse it and,
+assuming the organization exists and the calling user can access it, continue the request with the specified ID as its
+target organization. If the header is not found, it will fall back to one of the other methods described below.
+If the header is present and the user does not have access to the organization, the API will return a 404 status code.
+
+### Session-Based Organization Tracking
+
+If no organization-specifying request header is included in the request, the platform will then use session information
+to determine one.
+
+Sessions enable users to establish a long-lived association with an organization that can be used to automatically scope
+all requests to a specific organization. Users can establish a session using the
+[PUT /organizations/current](https://docs.arthur.ai/api-documentation/v3-api-docs.html#tag/organizations/paths/~1organizations~1current/put)
+endpoint. Similarly, they can retrieve the organization from their current session with the
+[GET /organizations/current](https://docs.arthur.ai/api-documentation/v3-api-docs.html#tag/organizations/paths/~1organizations~1current/get)
+endpoint.
+
+By calling `PUT /organizations/current` with a target organization specified, a user receives an `_arthur_session`
+cookie that associates future requests with the target organization. Any future calls to `PUT /organizations/current`
+will return a new `_arthur_session` cookie to track your new session for whichever organization is specified in
+the `PUT` call.
+
+By calling `GET /organizations/current` with an `_arthur_session` cookie included in the request, users can retrieve the
+organization their current session is associated with.
+
+```{note}
+Sessions do not provide any access to Arthur on their own. They only keep track of the user's association with a 
+specific organization. Users must also have an access token to authenticate with the Arthur APIs. 
+Since sessions do not grant any access to the Arthur platform, they expire after 30 days to prevent 
+users from repeatedly having to specify an organization.
+```
+
+### Arthur SDK Client
+
+For SSO-authenticated users with access to more than one organization working with the Arthur SDK, the SDK client allows
+selecting a specific organization. _Note: SDK users with access to a single Arthur organization will automatically have
+their one organization set._
+
+When an SDK user is instantiating the
+[`ArthurAI` client](https://docs.arthur.ai/sdk/sdk_v3/apiref/arthurai.client.client.ArthurAI.html#arthurai.client.client.ArthurAI)
+, they can first specify which organization they want that client to be associated with by passing in that
+organization's ID as the `organization_id` optional parameter.
+
+```python
+connection = ArthurAI(organization_id=org_id: str, ...)
+```
+
+After the `ArthurAI` client has been instantiated, users can update the client's current organization by calling
+the [`set_current_org()`](https://docs.arthur.ai/sdk/sdk_v3/apiref/arthurai.client.client.ArthurAI.html#arthurai.client.client.ArthurAI.set_current_org)
+method.
+
+```python
+connection.set_current_org(org_id: str)
+```
+
+At any point, SDK users can access their `ArthurAI` client's current organization by calling
+the [`get_current_org()`](https://docs.arthur.ai/sdk/sdk_v3/apiref/arthurai.client.client.ArthurAI.html#arthurai.client.client.ArthurAI.get_current_org)
+method.
+
+```python
+current_org = connection.get_current_org()
+``` 
+
+### Deep Linking to an Organization in the UI
+
+If an SSO-authenticated user has access to more than one organization, Arthur allows them to bypass manual organization
+selection through deep linking. A user can append the url parameter `?organization_id=<ORG-ID>` to the Arthur
+platform host name, where `<ORG-ID>` is equal to one of the organization IDs to which the user has access. This
+will tie the users session to that organization ID upon authentication through SSO.
+
+N.B. This will only work when appended to the platform host name.
+E.g. `https://[Arthur hostname]/?organization_id=<ORG-ID>` This will not work if you are attempting to apply the
+parameter to any other path on the Arthur host. E.g. `https://[Arthur hostname]/login/?organization_id=<ORG-ID>` will
+NOT work.
+
+```{toctree}
+:maxdepth: 2
+:hidden:
+
+OIDC <setting_up_oidc_idp>
+SAML <setting_up_saml_idp>
+Custom RBAC <custom_rbac>
+```
+

files/arthur-docs-markdown/platform-management/access-control-overview/sso-access-control/setting_up_oidc_idp.md.txt

@@ -0,0 +1,555 @@
+# Setting Up an OIDC IdP
+
+```{note}
+SSO configurations are only supported in on-prem Arthur installations.
+```
+
+This page provides a walk-through for how to configure your Arthur installation to work with an OIDC compatible IdP.
+In order to complete this guide, you need
+administrator access to your IdP and access to your Arthur installation's admin console configuration. Additionally, you
+will either need access to the Arthur superadmin user or be able to assume a role in your IdP to give yourself RBAC
+management permissions in Arthur.
+
+This guide will walk through the following steps:
+
+1. Configure the IdP user groups and OIDC token claims
+2. Configure the IdP OIDC Settings
+3. Configure Arthur to work with your IdP
+4. Apply the Arthur IdP YAML configuration
+5. Create organization user roles to match the IdP user groups
+6. Test Access
+7. Cleaning Up
+
+## 1. Configure the IdP user groups and OIDC token claims
+
+In order to properly map users to permissions, Arthur requires a claim in your OIDC JSON Web Token (JWT) that contains
+information about the group memberships of the user. Each group in the IdP should correspond to a role in Arthur's
+{doc}`custom_rbac` permission system.
+
+This process can vary depending on your IdP, but most IdP's should have a user grouping mechanism, and a mechanism to
+configure attributes in the JWT claims. For example using Okta, admins can
+configure the JWT claims to include group information under their account's Security -> API -> default -> Claims then
+the "Add Claim" button. From the popup, give the claim a name, "Groups" in this example, set the "Include in token type"
+to "Access Token", select "Value type" as "Groups", and include a "Matches regex" filter to select the groups to
+include:
+
+![okta-oidc-groups-claims](/_static/images/sso/okta-configure-oidc-groups-claims.png "Okta OIDC Groups Claims")
+
+Retrieving an example OIDC token is IdP-specific and may involve completing the sign-in flow via a script or API client.
+An alternative can be to use a 3rd party like [https://oidcdebugger.com](https://oidcdebugger.com), but that will
+require
+whitelisting `https://oidcdebugger.com/debug` as a valid redirect URL for your Arthur SSO app (this could be enabled
+temporarily for debugging, then removed). Here is an example JWT after setting the group claims field:
+
+```json
+{
+  "iss": "https://dev.okta.com/oauth2/default",
+  "aud": "api://default",
+  "scp": [
+    "openid"
+  ],
+  "Groups": [
+    "idp-admin",
+    "org-1-model-owner",
+    "org-2-model-owner"
+  ],
+  "FirstName": "John",
+  "LastName": "Doe",
+  "Login": "john.doe@arthur.ai"
+}
+```
+
+As the example token shows, the user's groups in the IdP are populated as a string list in the "Groups" field in the
+token. Arthur will use this list of groups to match the user to the corresponding roles in Arthur by name.
+
+## 2. Configure the IdP OIDC Settings
+
+In order for your IdP to speak to Arthur, it needs to know where to find it. Enter the following URL in your IdP's
+configuration to whitelist Arthur's callback endpoint (sign-in redirect
+URL): `https://<YOUR ARTHUR HOST>/login/callback`.
+
+Additionally, the IdP will need to know what OIDC protocol to speak with Arthur. Today, Arthur supports two protocol
+flows:
+
+1. `implicit`
+2. `PKCE`
+
+Both flows are intended to be used with "Single Page Applications" or SPAs. Follow the configuration for your IdP that
+matches one of those two flows with SPAs. Additionally, note the following settings from your IdP in order to use in
+the Arthur configuration below:
+
+- Client ID
+- Resource ID (if available)
+- OIDC flow (PKCE or implicit)
+- Audience (value that is set in the token by the IdP)
+
+## 3. Configure Arthur to work with your IdP
+
+Next, Arthur needs to know how to handshake with your IdP. To do that, Arthur requires the following
+information:
+
+1. Your IdP's discovery URL, typically in the format `<IdP path>/.well-known/openid-configuration` URL that contains the
+   relevant endpoints for your IdP.
+   ```{note}
+   If this page isn't accessible to Arthur due to CORS or other restrictions, the values can be provided manually.
+   ```
+2. One or more IdP administrator user groups that will be paired to global custom roles in Arthur
+   (see [here](#configure-the-arthur-global-roles) for a description of what these are for)
+3. An understanding of your OIDC token claims (values) and how to parse user information out of it
+4. The four configuration values captured above from your IdP
+
+With that information available, it is possible to fill out Arthur's IdP configuration YAML. The next subsections
+explain each subsection of the Arthur YAML configuration, and is followed by some complete examples further down.
+
+### Configuring the IdPs discovery URL
+
+Almost all OIDC IdP's have accessible discovery URLs, but some do not provide CORS support for them, so their contents
+need to be filled out manually. To support either option, Arthur has two configurations that can be
+used, `discoveryBaseURL` or `endpointOverrides`. If your IdP has CORS restrictions,
+see [Appendix B](#appendix-b-setup-for-idps-with-cors-restrictions) for additional
+setup that is required.
+
+```yaml
+# use this option if your IdP has an accessible discovery URL
+# IMPORTANT: don't include the /.well-known/openid-configuration suffix!!
+# for example, if the full URL is https://<HOST>/oauth2/default/.well-known/openid-configuration
+# only specify: https://<HOST>/oauth2/default
+discoveryBaseURL: "https://<HOST>/oauth2/default"
+# use this option if your IdP has CORS restrictions on the discovery URL, otherwise comment this out.
+# fill in the values manually from the discovery endpoint's contents
+endpointOverrides:
+  issuer: "issuer string for the IDP"
+  authorization_endpoint: "URL ending in /authorize"
+  token_endpoint: "URL ending in /token"
+  jwks_uri: "URL ending in /keys"
+  userinfo_endpoint: "URL ending in /userinfo"
+  # note not all IdPs will have the following endpoints, fill in as many as you can
+  end_session_endpoint: "URL ending in /logout"
+  device_authorization_endpoint: "URL ending in /devicecode"
+  revocation_endpoint: "URL ending in /revoke"
+  introspection_endpoint: "URL ending in /introspect"
+  registration_endpoint: "URL ending in /clients"
+```
+
+### Configure the Arthur Global Roles
+
+Arthur has the ability to create roles for the cluster administrators during the configuration of the IdP. These roles
+are often needed by admins to configure RBAC and create organizations for other users in the system. See
+{ref}`creating_global_roles_in_arthur_config` for a deep dive on how to use global roles.
+
+This section of the YAML config is under the `globalRoleDefs` field. It accepts a list of role definitions that will be
+created when the configuration is applied. The names of the roles in this section must match the user groups in your IdP
+in order to be able to assume them in Arthur.
+
+```yaml
+  globalRoleDefs:
+    # Here we can specify a list to define multiple global roles
+    - name: "idp-admin" # change this name to match the cluster administrator group name in your IdP
+      permissions:
+        custom_roles:
+          - read
+          - write
+          - delete
+        organization_global:
+          - read
+          - write
+        organization:
+          - read
+          - delete
+        model:
+          - read
+          - write
+          - delete
+```
+
+### Parsing the IdP JWT claims
+
+In order for Arthur to communicate with your IdP, it needs to understand the format of the JWT claims your IdP uses.
+This section of the config falls under the `accessTokenValidation` YAML field. This section is designed to be flexible
+to
+support a variety of claim formats, so it has a lot of options. At its core, the goal is to tell Arthur how to be
+able to extract the following information from the claims:
+
+- user roles/groups
+- first name
+- last name
+- email
+- user ID
+
+Each field has a corresponding YAML configuration that defines where to find the information in the JWT claims. For
+example:
+
+```yaml
+claimMapping:
+  firstName: FirstName
+```
+
+This configuration tells Arthur that it can find the user's first name under the "FirstName" claim in the
+JWT. Such a token might look like this:
+
+```json
+{
+  "iss": "https://dev.okta.com/oauth2/default",
+  "aud": "api://default",
+  "scp": [
+    "openid"
+  ],
+  "Groups": [
+    "idp-admin",
+    "org-1-model-owner",
+    "org-2-model-owner"
+  ],
+  "FirstName": "John",
+  "LastName": "Doe",
+  "Login": "john.doe@arthur.ai"
+}
+```
+
+More examples of how to parse user information out of the JWT claims can be
+found [below](#appendix-a-more-examples-of-jwt-claims-and-how-to-parse-them).
+
+### Full Configuration Examples
+
+Here is an example of a full configuration, combining each section described above.
+
+```yaml
+version: v2
+kind: OIDC
+config:
+  # discovery URL without the /.well-known/openid-configuration suffix
+  discoveryBaseURL: https://example.com/oauth2/default
+  # Either "implicit" or "PKCE"
+  flowType: PKCE
+  # client ID from your IdP for the Arthur SSO application
+  clientID: "client id string"
+  # optional: resource ID from your IdP for the Arthur SSO application if required by the IdP
+  resourceID: ""
+  authorizeScopes:
+    - openid  # required for OIDC
+  # use this section to define global roles
+  # one example role would be to give the cluster admin permissions to create organizations and manage custom roles
+  globalRoleDefs:
+    - name: "iam-admin"  # change this to match the user group name in your IdP for administrators
+      permissions:
+        custom_roles:
+          - read
+          - write
+          - delete
+        organization_global:
+          - read
+          - write
+        organization:
+          - read
+          - delete
+  # this section describes how to parse the user information out of the JWT returned from the IdP
+  # this is used by Arthur to understand who the user is and what their roles are
+  accessTokenValidation:
+    type: JWT  # only JWT is supported today
+    # fields in the token Arthur will use to extract the authentication information
+    claimMapping:
+      roles: Groups  # this is telling Arthur to look in the "Groups" claim to find the list of user's roles
+      userID: EmployeeID
+      username: Login
+      firstName: FirstName
+      lastName: LastName
+      email: Login
+
+    # one or more audiences to validate, this should match your IdP's configuration
+    audience:
+      - api://default
+
+    # optional override signature algo
+    # signatureAlgo: RS256
+
+```
+
+Here is an additional descriptions of the fields that need to be set in the config YAML above:
+
+* `discoveryBaseURL`: This is the base URL for your Identity Provider. Your IdP should have
+  a `/.well-known/openid-configuration` endpoint and the
+  discoveryBaseURL is simply that url minus the `/.well-known/openid-configuration` part.
+* `flowType`: We support both implicit and PKCE flows. Consult with your team to decide which OIDC flow type is right
+  for your organization.
+* `clientID`: When you create the application integration in your IdP, a Client ID will be provided to you. Paste that
+  value into this field.
+* `resourceID`: This is optional. If your IdP gives you a Resource ID when creating your application integration, paste
+  the value here.
+* `claimMapping`: We extract various pieces of authentication information from the provided JWT access token. However,
+  there is no common standard for
+  how these pieces of information should be formatted in the token. For us to extract this information from the token,
+  we need you to explicitly tell Arthur
+  where this information is stored in the token. For example, a username could be stored in a field called `username`
+  or `login` or `email` or `userID`.
+  In order to get this user information, a mapping needs to be provided for the following items
+    * `roles`: This is the field for where either a single authorization role or a list of authorization roles will be
+      specified.
+      Note that this is **not** where you paste a list of roles, this is **the name of a field** in the JWT where the
+      user's roles are specified.
+      For help with role definitions, see {doc}`custom_rbac`.(Required)
+    * `userID`: This is the field for a unique identifier for the user; this is frequently the same as `username`
+      and/or `email`. (Optional, omit if empty)
+    * `username`: This is the field for the user's unique username; this is frequently the same as `username`
+      and/or `email`. (Optional, omit if empty)
+    * `firstName`: This is the field for the user's first name. (Optional, omit if empty)
+    * `lastName`: This is the field for the user's last name. (Optional, omit if empty)
+    * `email`: This is the field for the user's email. (Optional, omit if empty)
+* `audience`: This is part of the JWT standard. The `aud` field for any JWT you create must be a value in this list. For
+  example in the above configuration, any token that has an `aud` field that is not set to `api://defaults`, the token
+  will be automatically rejected by Arthur. If you are having trouble finding this value, it is frequently the same as
+  your `resourceID`. Remember to format this as a _list_, not a single value.
+
+```{note}
+If your IdP has CORS restrictions see
+[Appendix B](#appendix-b-setup-for-idps-with-cors-restrictions) below for a workaround.
+```
+
+## 4. Apply the Arthur IdP YAML configuration
+
+Once you have your YAML configuration file ready, you need to add it to your Arthur installation. With the Arthur admin
+console open, navigate to the "Use a 3rd Party Global Identity Provider" section and select "OIDC". This will expose a
+text box for you to paste the YAML config file assembled above. When pasting, make sure whitespace is preserved and the
+YAML document has consistent spacing (do not mix tabs and spaces). Here is a screenshot of the config section:
+
+![arthur-oidc-config-page](/_static/images/sso/arthur-oidc-config-page.png "Arthur OIDC Configuration Page")
+
+Once you have added your config files, scroll to the bottom and click "Save" to save the config. Then go to the latest
+version and click "Deploy" to roll out the change to the cluster.
+
+## 5. Create organization user roles to match the IdP user groups
+
+In order to complete this section, you will need access to the Arthur superadmin user credentials set during your
+install, or you will need to be able to assume the role defined in the Arthur IdP config YAML created above in
+the `globalRoleDefs` section.
+
+In order to use the API example linked below, you will need a Bearer token (authentication token) to include with your
+API request. There are a few options available to retrieve a token:
+
+1. **Retrieve a global role token directly from your IdP** - Most IdPs will have a method to retrieve tokens for users.
+   Some companies make scripts or APIs that allow retrieving a token. If your IdP does not have an automated method to
+   retrieve a token, you can try setting up a tool like [https://oidcdebugger.com](https://oidcdebugger.com) (this may
+   involve adding `https://oidcdebugger.com` as an allowed URL in your IdP settings).
+2. **Retrieve a global role token from your browser cookies** - If you sign in to Arthur as a user with a global role,
+   the UI will not be fully functional, but it will have a valid access token in the cookies. If you navigate to your
+   browser's developer console and then go to the Application Storage/Cookies section, you should see a cookie like
+   `ArthurAuth0`, which is your authentication token. Note: if your user has a large number of groups, there may be
+   multiple cookies of the form `ArthurAuthN`. In this case your token was too large to fit in the browser cookie, so
+   it had to be split. You can assemble the full token by concatenating the values of the `ArthurAuthN` cookies in
+   order.
+3. Use the [/login](https://docs.arthur.ai/api-documentation/v3-api-docs.html#tag/auth/paths/~1login/post) API endpoint
+   with the superadmin user's credentials set during the Arthur install (only available on-prem).
+
+Using either of those credentials, we can use the Arthur API to define roles in Arthur that match the user group names
+in your IdP. See the {ref}`creating_organization_roles` section for an example API request to create custom roles in
+Arthur. Importantly, the role
+names must uniquely match to a user group in your IdP in order for your users to be able to assume those permissions in
+Arthur. Therefore, the roles in Arthur must be globally unique in the entire Arthur installation.
+
+## 6. Test Access
+
+At this point everything should be configured correctly to sign in to Arthur via SSO. Either navigate to your IdP
+or the Arthur homepage to test logging in.
+
+## 7. Cleaning Up
+
+Once users are successfully able to log in to Arthur via the IdP, you should do the following to ensure proper security
+best-practices remain enforced:
+
+- Restrict any Arthur global roles to only allow access to essential admin functions
+- Set the Arthur superadmin user password securely, and either store the password in a vault, or discard the password
+  entirely. superadmin shouldn't be used going forward.
+- Set up a policy to routinely rotate the superadmin password to keep it secure
+
+Together, these practices will help ensure the security of your Arthur installation, and will give your IdP sole control
+over the platform and who is able to access it.
+
+## Common Troubleshooting
+
+If after following the steps above, users are not able to log in via the IdP try some of these common troubleshooting
+tips:
+
+### Does the user properly redirected to the IdP's log in screen?
+
+If not, there is likely a configuration error in the Arthur YAML config with the IdP discovery URL. Double check
+that the url entered resolved correctly when you append `/.well-known/openid-configuration` to the end of it. The full
+URL should be viewable in your browser or via a REST client.
+
+### Once the user authenticates with the IdP, are they redirected to the Arthur homepage?
+
+If not, there is likely a configuration error with the IdP and the URLs that it uses to communicate with Arthur.
+Double-check the redirect (whitelisted) URL is configured correctly for the Arthur installation
+at `https://<HOSTNAME>/login/callback`.
+
+### A user can see the Arthur home page, but can't see any of the model in their organization
+
+If a user cannot see any of the models in their organization, it means they either don't have the necessary permissions
+to access models (see {doc}`../../reference/permissions_by_endpoint`) or they were not able to correctly assume the role
+in Arthur. Double-check the groups in
+their JWT claims match the role names that have been configured in Arthur. A superadmin or global role user with
+permissions to manage RBAC can see a list of roles in the installation by using the following API call. Be sure to
+replace the HOST and AUTH TOKEN for your installation and user:
+
+```shell
+curl --location 'https://<HOST>/api/v3/authorization/custom_roles' \
+--header 'Authorization: Bearer <INSERT AUTH TOKEN HERE>'
+```
+
+## Appendix A: More examples of JWT claims and how to parse them
+
+This section outlines some additional ways to use the `accessTokenValidation` section of the Arthur IdP config YAML
+format. The below examples include sample JWTs, then corresponding YAML for how to parse them.
+
+### Basic Full Example
+
+This example shows how to parse a user's information from JWT claims in a typical format.
+Example parsed JWT claims JSON:
+
+```json
+{
+  "iss": "https://dev.okta.com/oauth2/default",
+  "aud": "api://default",
+  "scp": [
+    "openid"
+  ],
+  "Groups": [
+    "idp-admin",
+    "org-1-model-owner",
+    "org-2-model-owner"
+  ],
+  "FirstName": "John",
+  "LastName": "Doe",
+  "Login": "john.doe@arthur.ai",
+  "EmployeeID": "1234567890"
+}
+```
+
+Corresponding settings for the Arthur IdP config YAML `accessTokenValidation` for the user information field:
+
+```yaml
+accessTokenValidation:
+  type: JWT
+  claimMapping:
+    roles: Groups
+    userID: EmployeeID
+    username: Login
+    firstName: FirstName
+    lastName: LastName
+    email: Login
+```
+
+### Minimal Example
+
+This example shows how to parse a user's information from JWT claims when many fields are missing.
+Example parsed JWT claims JSON:
+
+```json
+{
+  "iss": "https://dev.okta.com/oauth2/default",
+  "aud": "api://default",
+  "scp": [
+    "openid"
+  ],
+  "Groups": [
+    "idp-admin",
+    "org-1-model-owner",
+    "org-2-model-owner"
+  ],
+  "user": "john.doe@arthur.ai"
+}
+```
+
+Corresponding settings for the Arthur IdP config YAML `accessTokenValidation` for the user information field:
+
+```yaml
+accessTokenValidation:
+  type: JWT
+  claimMapping:
+    roles: Groups
+    userID: user
+    username: user
+    firstName: ""
+    lastName: ""
+    email: user
+```
+
+## Appendix B: Setup for IdPs with CORS Restrictions
+
+Completing this will require access to the Kubernetes cluster Arthur is running in, and the ability to create ingress
+resources in that cluster.
+
+If your OIDC Identity Provider does not support CORS (common with Microsoft Azure AD), you will need to proxy
+requests via the Arthur backend. The following examples show how this can be done with a cluster using the
+NGINX ingress controller.
+
+This first example YAML configures a route on NGINX that will proxy OIDC connections to your IdP. You'll need to
+replace the `<IDP HOST>` and `<ARTHUR HOST>` placeholders, then apply it to your cluster with
+`kubectl apply -n <NAMESPACE> -f file.yaml`. There should be two places to fill in each variable below.
+
+```yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: external-idp
+spec:
+  type: ExternalName
+  externalName: "<IDP HOST>"
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: external-idp
+  annotations:
+    kubernetes.io/ingress.class: nginx
+    nginx.ingress.kubernetes.io/rewrite-target: /$2
+    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" #important
+    nginx.ingress.kubernetes.io/upstream-vhost: "<IDP HOST>"
+spec:
+  rules:
+    - host: "<ARTHUR HOST>"
+      http:
+        paths:
+          - backend:
+              service:
+                name: external-idp
+                port:
+                  number: 443
+            path: /oidc(/|$)(.*)
+            pathType: Prefix
+  tls:
+    - hosts:
+        - "<ARTHUR HOST>"
+      secretName: kotsadm-tls
+```
+
+After you've applied the above configuration to your cluster, you should be able to visit your IdP's
+`/.well-known/openid-configuration` endpoint at the following URL:
+`https://<ARTHUR HOST>/oidc/<your IdP's .well-known endpoint path>`. Once that is accessible, we need to modify
+the OIDC YAML configuration file. Fill in the following example
+with the correct values in the `endpointOverrides` section. Note, the `issuer` and `authorization_endpoint` fields
+should match what is in your IdP's `/.well-known` spec. The rest of the values should use the same path as shown in the
+IdP's `/.well-known` spec, but with the value of `<ARTHUR HOST>/oidc/` substituted for the host of the IdP.
+The following example shows a proper CORS config for an IdP at the `https://XXXX.okta.com` address.
+
+```yaml
+version: v2
+kind: OIDC
+config:
+  discoveryBaseURL: https://XXXX.okta.com/oauth2/default
+  # if your IdP has CORS restrictions with the metadata URL,
+  # specify this block to prevent using the metadata endpoint to look them up
+  endpointOverrides:
+    # these first two match the IdP's .well-known spec 
+    issuer: "https://XXXX.okta.com/oauth2/default"
+    authorization_endpoint: "https://XXXX.okta.com/oauth2/default/authorize"
+    # notice the following are all modified to add the <ARTHUR HOST>/oidc prefix in the URL
+    token_endpoint: "https://<ARTHUR HOST/oidc/oauth2/default/tokens"
+    jwks_uri: "https://<ARTHUR HOST/oidc/oauth2/default/keys"
+    userinfo_endpoint: "https://<ARTHUR HOST/oidc/oauth2/default/user_info"
+    end_session_endpoint: "https://<ARTHUR HOST/oidc/oauth2/default/logout"
+
+  # the rest of this file is unchanged from the examples above...
+```
+
+Once you have modified this YAML file accordingly, follow the [steps](#apply-the-arthur-idp-yaml-configuration)
+above to save it to your installation.
+
+

files/arthur-docs-markdown/platform-management/access-control-overview/sso-access-control/setting_up_saml_idp.md.txt

@@ -0,0 +1,479 @@
+# Setting Up a SAML IdP
+
+```{note}
+SSO configurations are only supported in on-prem Arthur installations. 
+```
+
+This page provides a walk-through for how to configure your Arthur installation to work with a SAML compatible IdP.
+In order to complete this guide, you need
+administrator access to your IdP and access to your Arthur installation's admin console configuration. Additionally, you
+will either need access to the Arthur superadmin user or be able to assume a role in your IdP to give yourself RBAC
+management permissions in Arthur.
+
+This guide will walk through the following steps:
+
+1. Configure the IdP user groups and SAML assertion
+2. Configure the Arthur service provider URLs in the IdP
+3. Configure Arthur to work with your IdP
+4. Apply the Arthur IdP YAML configuration
+5. Create organization user roles to match the IdP user groups
+6. Test Access
+7. Cleaning Up
+
+## 1. Configure the IdP user groups and SAML assertion
+
+In order to properly map users to permissions, Arthur requires an attribute in your SAML assertion that contains
+information about the group memberships of the user. Each group in the IdP should correspond to a role in Arthur's
+{doc}`custom_rbac` permission system.
+
+This process can vary depending on your IdP, but most IdP's should have a user grouping mechanism, and a mechanism to
+configure attributes in the SAML assertions. For example using Okta, under the SAML application settings, admins can
+configure the SAML assertion attributes to include group information under SAML Settings -> Configure SAML -> Group
+Attribute Statements, then specifying a name for the attribute and a filter for the groups to include:
+
+![okta-saml-groups-attribute](/_static/images/sso/okta-configure-saml-groups-assertion.png "Okta SAML Assertion Groups Attribute")
+
+Setting this configuration produces the following attribute in the SAML assertion (in Okta click "Preview the SAML
+Assertion" button to see a sample):
+
+```xml
+
+<saml2:AttributeStatement>
+    <saml2:Attribute Name="groups">
+        <saml2:AttributeValue>Everyone</saml2:AttributeValue>
+        <saml2:AttributeValue>admins</saml2:AttributeValue>
+        <saml2:AttributeValue>org-1-model-owners</saml2:AttributeValue>
+    </saml2:Attribute>
+</saml2:AttributeStatement>
+
+```
+
+## 2. Configure the Arthur service provider URLs in the IdP
+
+In order for your IdP to speak to Arthur, it needs to know where to find it. Enter the following URLs in your IdP's
+configuration to Arthur's SAML endpoints:
+
+- ACS URL (SSO URL): `https://<HOSTNAME>/api/v3/saml/sso`
+- Entity ID: `https://<HOSTNAME>/api/v3/saml/sso`
+- Start URL: `https://<HOSTNAME>/`
+
+```{note}
+If your IdP will be sending signed assertions to Arthur, you will also need to generate and upload the public key 
+(certificate) Arthur will be using in your IdP. This will be the same certificate you set in the Arthur configuration
+[below](#apply-the-arthur-idp-yaml-configuration). Please follow your own company policies to obtain a certificate 
+for Arthur. If you have no internal guidelines, then use a tool like 
+[ssh-keygen](https://www.ibm.com/docs/en/spectrum-control/5.3.7?topic=agents-creating-certificate-ssh-protocol) to
+generate them
+```
+
+## 3. Configure Arthur to work with your IdP
+
+Additionally, Arthur needs to know how to handshake with your IdP. To do that, Arthur requires the following
+information:
+
+1. Your IdP's metadata URL or the metadata XML payload (some IdPs require it be downloaded, either is fine)
+2. One or more IdP administrator user groups that will be paired to global custom roles in Arthur
+   (see [here](#configure-the-arthur-global-roles) for a description of what these are for)
+3. An understanding of your SAML assertion and how to parse user information out of it
+
+With those three things available, it is possible to fill out Arthur's IdP configuration YAML. The next subsections
+explain each section of the Arthur YAML configuration, and are followed by some complete examples further down.
+
+### Configuring the IdPs metadata URL
+
+Some IdP's host their metadata XML at a public URL, while others only have it available for download privately. To
+support either option, Arthur has two configurations that can be used:
+
+```yaml
+# use this option if your IdP has a public URL for its metadata
+metadataURL: "link to IdP metadata goes here"
+# use this option if your IdP does not have a public URL and include the XML payload
+# make sure to indent the XML payload two spaces and make sure the X509Certificate lines 
+# do not have more than two leading whitespaces!
+metadataXML: |
+  <?xml version="1.0" encoding="UTF-8"?><md:EntityDescriptor ...>
+    <md:IDPSSODescriptor>
+      ...
+        <ds:X509Certificate>CERTIFICATE LINE 1
+  CERT LINE 2
+  CERT LINE 3
+  CERT LINE 4
+  LAST CERT LINE</ds:X509Certificate>
+    </md:IDPSSODescriptor>
+  </md:EntityDescriptor>
+```
+
+```{warning}
+If using the `metadataXML` configuration option, make sure to indent the entire XML payload two spaces. YAML expects 
+multi-line values to be indented under the key `metadataXML`.
+```
+
+```{warning}
+Additionally, the assertion's `X509Certificate` XML attribute is a multi-line value within the XML. 
+Any new lines in the certificate value need to be indented only two spaces (all the way to the left of the YAML value). 
+Otherwise, the extra whitespaces introduces characters which will invalidate the certificate value.
+```
+
+### Configure the Arthur Global Roles
+
+Arthur has the ability to create roles for the cluster administrators during the configuration of the IdP. These roles
+are often needed by admins to configure RBAC and create organizations for other users in the system. See
+{ref}`creating_global_roles_in_arthur_config` for a deep dive on how to use global roles.
+
+This section of the YAML config is under the `globalRoleDefs` field. It accepts a list of role definitions that will be
+created when the configuration is applied. The names of the roles in this section must match the user groups in your IdP
+in order to be able to assume them in Arthur.
+
+```yaml
+  globalRoleDefs:
+    # Here we can specify a list to define multiple global roles
+    - name: "idp-admin" # change this name to match the cluster administrator group name in your IdP
+      permissions:
+        custom_roles:
+          - read
+          - write
+          - delete
+        organization_global:
+          - read
+          - write
+        organization:
+          - read
+          - delete
+        model:
+          - read
+          - write
+          - delete
+```
+
+### Parsing the IdP SAML Assertion
+
+In order for Arthur to communicate with your IdP, it needs to understand the format of the SAML assertion your IdP uses.
+This section of the config falls under the `assertionAttributes` YAML field. This section is designed to be flexible to
+support a variety of assertion formats, so it has a lot of options. At its core, the goal is to tell Arthur how to be
+able to extract the following information from the assertion:
+
+- user roles/groups
+- first name
+- last name
+- email
+- user ID
+
+Each field has a corresponding YAML field the defines where to find the information in the SAML assertion XML. For
+example:
+
+```yaml
+firstNameAttribute:
+  name: "employeeFirstName"
+```
+
+This configuration tells Arthur that it can find the user's first name under the "employeeFirstName" attribute in the
+XML assertion. Such an assertion might look like this:
+
+```xml
+
+<saml2:AttributeStatement>
+    <saml2:Attribute Name="firstName">
+        <saml2:AttributeValue>Ian</saml2:AttributeValue>
+    </saml2:Attribute>
+</saml2:AttributeStatement>
+```
+
+More examples of how to parse attributes out of the SAML assertion can be
+found [below](#appendix-a-more-examples-of-saml-assertion-values-and-how-to-parse-them).
+
+### Full Configuration Examples
+
+Here is an example of a full configuration, combining each section described above.
+
+```yaml
+version: v1
+kind: SAML
+config:
+  # if your IdP hosts its metadata, provide the URL to it here
+  metadataURL: "link to IdP metadata goes here"
+  # if the IdP does not host the metadata, provide the XML payload here and comment out metadataURL
+  #  metadataXML: |
+  #     <?xml version="1.0" encoding="UTF-8"?><md:EntityDescriptor ...>
+  #       <md:IDPSSODescriptor>
+  #         ...
+  #       </md:IDPSSODescriptor>
+  #     </md:EntityDescriptor>
+
+  # this section describes how Arthur will parse the SAML assertion from your IdP
+  # for each required attribute, Arthur will use the "name" field to match to an XML attribute in the SAML assertion
+  assertionAttributes:
+    # this roleAttribute configuration will use a "groups" attribute in the XML assertion which expects the 
+    # roles in separate XML AttributeValues within the assertion Attribute
+    roleAttribute:
+      name: "groups"
+      useAllAttributeValues: True
+    firstNameAttribute:
+      name: "employeeFirstName"
+    lastNameAttribute:
+      name: "employeeLastName"
+    emailAttribute:
+      name: "company_email"
+    userIdAttribute:
+      name: "companyUserID"
+  globalRoleDefs:
+    # Here we specify a global role for the IdP user group "idp-admin" to create and manage RBAC in Arthur
+    - name: "idp-admin"
+      permissions:
+        custom_roles:
+          - read
+          - write
+          - delete
+        organization_global:
+          - read
+          - write
+        organization:
+          - read
+          - delete
+        model:
+          - read
+          - write
+          - delete
+
+```
+
+## 4. Apply the Arthur IdP YAML configuration
+
+Once you have your YAML configuration file ready, you need to add it to your Arthur installation. With the Arthur admin
+console open, navigate to the "Use a 3rd Party Global Identity Provider" section and select "SAML". This will expose a
+text box for you to paste the YAML config file assembled above. When pasting, make sure whitespace is preserved and the
+YAML document has consistent spacing (do not mix tabs and spaces). Here is a screenshot of the config section:
+
+![arthur-saml-config-page](/_static/images/sso/arthur-saml-config-page.png "Arthur SAML Configuration Page")
+
+```{note}
+If your IdP enforces signed authorization requests, this config page also provides the ability to upload a 
+certificate and private key for Arthur to use when making the requests. Click the "Upload a file" button for the Public 
+Certificate and Private Key sections of the config to upload the appropriate files for your IdP.
+```
+
+Once you have added your config files, scroll to the bottom and click "Save" to save the config. Then go to the latest
+version and click "Deploy" to roll out the change to the cluster.
+
+## 5. Create organization user roles to match the IdP user groups
+
+In order to complete this section, you will need access to the Arthur superadmin user credentials set during your
+install, or you will need to be able to assume the role defined in the Arthur IdP config YAML created above in
+the `globalRoleDefs` section.
+
+In order to use the API example linked below, you will need a Bearer token (authentication token) to include with your
+API request. There are a few options available to retrieve a token:
+
+1. **Retrieve a SAML assertion from your IdP and exchange with Arthur** - Most IdPs will have a method to retrieve a
+   SAML assertion for users. Some companies make scripts or APIs to do so. If your IdP does not have an automated method
+   to retrieve an assertion, use one of the other options below. Once you have an assertion, you can exchange it for an
+   Arthur access token with the follow API call to Arthur:
+   ```shell
+   curl --location --request POST 'https://<YOUR ARTHUR HOST>/api/v3/saml/sso' \
+   --header 'Content-Type: application/x-www-form-urlencoded' \
+   --data-urlencode 'SAMLResponse=<INSERT URL ENCODED BASE64 ASSERTION>'
+   ```
+2. **Retrieve a global role token from your browser cookies** - If you sign in to Arthur as a user with a global role,
+   the UI will not be fully functional, but it will have a valid access token in the cookies. If you navigate to your
+   browser's developer console and then go to the Application Storage/Cookies section, you should see a cookie like
+   `Authentication`. The authentication token is the value of that cookie.
+3. Use the [/login](https://docs.arthur.ai/api-documentation/v3-api-docs.html#tag/auth/paths/~1login/post) API endpoint
+   with the superadmin user's credentials set during the Arthur install (only available on-prem).
+
+Using either of those credentials, we can use the Arthur API to define roles in Arthur that match the user group names
+in your IdP. See the {ref}`creating_organization_roles` section for an example API request to create custom roles in
+Arthur. Importantly, the role
+names must uniquely match to a user group in your IdP in order for your users to be able to assume those permissions in
+Arthur. Therefore, the roles in Arthur must be globally unique in the entire Arthur installation.
+
+## 6. Test Access
+
+At this point everything should be configured correctly to sign in to Arthur via SSO. Either navigate to your IdP
+or the Arthur homepage to test logging in.
+
+## 7. Cleaning Up
+
+Once users are successfully able to log in to Arthur via the IdP, you should do the following to ensure proper security
+best-practices remain enforced:
+
+- Restrict any Arthur global roles to only allow access to essential admin functions
+- Set the Arthur superadmin user password securely, and either store the password in a vault, or discard the password
+  entirely. superadmin shouldn't be used going forward.
+- Set up a policy to routinely rotate the superadmin password to keep it secure
+
+Together, these practices will help ensure the security of your Arthur installation, and will give your IdP sole control
+over the platform and who is able to access it.
+
+## Common Troubleshooting
+
+If after following the steps above, users are not able to log in via the IdP try some of these common troubleshooting
+tips:
+
+### Does the user properly redirected to the IdP's log in screen?
+
+If not, there is likely a configuration error in the Arthur YAML config with the IdP metadata or the URL to access it.
+Another problem could be if your IdP expects Arthur to make signed requests to authenticate users. If that is the case,
+be sure you have correctly configured Arthur's certificate and private key as
+described [above](#apply-the-arthur-idp-yaml-configuration).
+
+### Once the user authenticates with the IdP, are they redirected to the Arthur homepage?
+
+If not, there is likely a configuration error with the IdP and the URLs that it uses to communicate with Arthur.
+Double-check the ACS (SSO) URL is configured correctly for the Arthur installation
+at `https://<HOSTNAME>/api/v3/saml/sso`.
+
+### A user can see the Arthur home page, but can't see any of the model in their organization
+
+If a user cannot see any of the models in their organization, it means they either don't have the necessary permissions
+to access models (see {doc}`../../reference/permissions_by_endpoint`) or they were not able to correctly assume the role
+in Arthur. Double-check the groups in
+their SAML assertion match the role names that have been configured in Arthur. A superadmin or global role user with
+permissions to manage RBAC can see a list of roles in the installation by using the following API call. Be sure to
+replace the HOST and AUTH TOKEN for your installation and user:
+
+```shell
+curl --location 'https://<HOST>/api/v3/authorization/custom_roles' \
+--header 'Authorization: Bearer <INSERT AUTH TOKEN HERE>'
+```
+
+## Appendix A: More examples of SAML assertion values and how to parse them
+
+This section outlines some additional ways to use the `assertionAttributes` section of the Arthur IdP config YAML
+format. The below examples include sample SAML assertions, then corresponding YAML for how to parse them.
+
+### Basic Full Example
+
+This example shows how to parse a user's information from a SAML assertion when each field is its own Attribute and the
+user's groups are each in their own AttributeValue.
+Example SAML assertion XML:
+
+```xml
+
+<saml2:AttributeStatement>
+    <saml:Attribute Name="employeeFirstName">
+        <saml:AttributeValue>John</saml:AttributeValue>
+    </saml:Attribute>
+    <saml:Attribute Name="employeeLastName">
+        <saml:AttributeValue>Doe</saml:AttributeValue>
+    </saml:Attribute>
+    <saml:Attribute Name="employeeEmail">
+        <saml:AttributeValue>john.doe@arthur.ai</saml:AttributeValue>
+    </saml:Attribute>
+    <saml:Attribute Name="employeeID">
+        <saml:AttributeValue>1234567890</saml:AttributeValue>
+    </saml:Attribute>
+    <saml:Attribute Name="userGroups">
+        <saml:AttributeValue>group1</saml:AttributeValue>
+        <saml:AttributeValue>group2</saml:AttributeValue>
+        <saml:AttributeValue>group3</saml:AttributeValue>
+    </saml:Attribute>
+</saml2:AttributeStatement>
+
+```
+
+Corresponding settings for the Arthur IdP config YAML `assertionAttributes` for the user information field:
+
+```yaml
+  assertionAttributes:
+    roleAttribute:
+      name: "userGroups"
+      useAllAttributeValues: True
+    firstNameAttribute:
+      name: "employeeFirstName"
+    lastNameAttribute:
+      name: "employeeLastName"
+    emailAttribute:
+      name: "employeeEmail"
+    userIdAttribute:
+      name: "employeeID"
+```
+
+### Parsing User Groups from Multiple Attribute Values
+
+This example shows how to parse a user's groups from a SAML assertion when each group is its own AttributeValue.
+Example SAML assertion XML:
+
+```xml
+
+<saml2:AttributeStatement>
+    <saml:Attribute Name="Idp_user_groups">
+        <saml:AttributeValue>role1</saml:AttributeValue>
+        <saml:AttributeValue>role2</saml:AttributeValue>
+        <saml:AttributeValue>role3</saml:AttributeValue>
+    </saml:Attribute>
+    ...
+</saml2:AttributeStatement>
+```
+
+Corresponding settings for the Arthur IdP config YAML `assertionAttributes` for the `roleAttribute` field:
+
+```yaml
+  assertionAttributes:
+    roleAttribute:
+      name: "Idp_user_groups"
+      useAllAttributeValues: True
+```
+
+### Parsing User Groups from a String Attribute Value
+
+This example shows how to parse a user's groups from a SAML assertion when the groups are in a single string
+AttributeValue.
+Example SAML assertion XML:
+
+```xml
+
+<saml2:AttributeStatement>
+    <saml:Attribute Name="Idp_user_groups">
+        <saml:AttributeValue>role1,role2,role3</saml:AttributeValue>
+    </saml:Attribute>
+    ...
+</saml2:AttributeStatement>
+```
+
+Corresponding settings for the Arthur IdP config YAML `assertionAttributes` for the `roleAttribute` field:
+
+```yaml
+  assertionAttributes:
+    roleAttribute:
+      name: "Idp_user_groups"
+      deliminator: ","
+```
+
+### Parsing Specific Fields in a Single Attribute's AttributeValue List
+
+This example shows how to parse a user's information from a SAML assertion when all fields are in a single assertion
+Attribute's list of AttributeValues.
+Example SAML assertion XML:
+
+```xml
+
+<saml2:AttributeStatement>
+    <saml:Attribute Name="employeeInfo">
+        <saml:AttributeValue>John</saml:AttributeValue>
+        <saml:AttributeValue>Doe</saml:AttributeValue>
+        <saml:AttributeValue>(123) 456-7890</saml:AttributeValue>
+        <saml:AttributeValue>42 Wallaby Way, Sydney</saml:AttributeValue>
+        <saml:AttributeValue>johndoe@arthur.ai</saml:AttributeValue>
+        <saml:AttributeValue>5678987654</saml:AttributeValue>
+    </saml:Attribute>
+    ...
+</saml2:AttributeStatement>
+```
+
+Corresponding settings for the Arthur IdP config YAML `assertionAttributes` for the user information field:
+
+```yaml
+  assertionAttributes:
+    firstNameAttribute:
+      name: "employeeInfo"
+      index: 0
+    lastNameAttribute:
+      name: "employeeInfo"
+      index: 1
+    emailAttribute:
+      name: "employeeInfo"
+      index: 4
+    userIdAttribute:
+      name: "employeeInfo"
+      index: 5
+```
+

files/arthur-docs-markdown/platform-management/access-control-overview/standard_access_control.md.txt

@@ -0,0 +1,76 @@
+# Arthur Standard Access Control Overview
+
+In both SaaS and On-prem installations, Arthur ships with a build-in access control system that can be used to manage
+users, permissions, and access to organizations. This system has different capabilities than the SSO based paradigm. If
+your installation is using SSO, please see the {doc}`sso-access-control/index`.
+
+## Authentication
+
+Users authenticate to Arthur using a username and password, which is set when their account is created and can be
+changed later in the UI. Users can also use
+the [login API endpoint](https://docs.arthur.ai/api-documentation/v3-api-docs.html#tag/auth/paths/~1login/post) to
+retrieve a token for use with Arthur APIs.
+
+Applications and automated systems can authenticate with Arthur using API keys, which can be created in the Arthur UI
+from the organization menu in the upper right corner then clicking on Manage API Keys.
+
+```{note}
+Note: it is not recommended to use API-keys for non-automated use cases as they are not tied to user 
+identities and can obscure who is performing actions. As a best practice, use API keys minimally only in the systems 
+that need automated access, and be sure to create a rotation practice to ensure safe keeping.
+```
+
+![manage-api-keys](/_static/images/manage-api-keys.png "Organization Menu")
+
+## Authorization (RBAC)
+
+The Arthur standard access control system uses role-based access control (RBAC) with a set of pre-defined roles.
+The available roles for users are `User`, `Model Owner`, `Administrator`, and `SuperAdmin`. If enrolled in multiple
+organizations, the user can have a different role in each organization. For a full list of permissions for these 4
+standard roles, please reference {doc}`here </platform-management/reference/permissions_by_standard_roles>`.
+
+* `User`: Has read-only access to the models and data within the organization.
+* `Model Owner`: Can onboard new models in the enrolled organization as well as send data including reference data,
+  inferences, and ground truth.
+* `Administrator`: Organization level administrator that has access to manage users and models within the organization.
+* `Super Admin`: Has full access to all data, models, and actions on the platform. Can create new organizations and
+  manage users. Only available on-prem.
+
+```{note} 
+If your installation uses SSO, you can take advantage of creating custom roles to fine-tune user 
+access to Arthur resources. See {doc}`sso-access-control/custom_rbac` for more information.
+```
+
+## Adding Users to an Organization in the UI
+
+To complete this section, you must have the "Administrator" role in your organization.
+
+In the upper right corner, click on the organization menu, then click "Manage Members". From this screen, you can enter
+the emails of additional users to add to the organization, manage the roles of existing users, and remove users from the
+organization.
+
+```{note}
+In order for email-based user invites to work, your installation must have an email integration set up. If
+not, you can use the
+[Arthur API](https://docs.arthur.ai/api-documentation/v3-api-docs.html#tag/users/paths/~1users/post)
+to create user accounts directly in your organization.
+```
+
+## Adding Users to an Organization in the API
+
+Arthur also supports managing users via automated workflows using the REST API. In order to create a user in
+your organization, you will need to have Administrator privileges in that organization, or have access to the superadmin
+user for your Arthur on-prem installation. The following APIs are helpful for managing users:
+
+- [Create a New User](https://docs.arthur.ai/api-documentation/v3-api-docs.html#tag/users/paths/~1users/post)
+- [Update User/Change User Password](https://docs.arthur.ai/api-documentation/v3-api-docs.html#tag/users/paths/~1users~1%7Buser_id%7D/patch)
+- [Send an Email Invite to a New User](https://docs.arthur.ai/api-documentation/v3-api-docs.html#tag/users/paths/~1users~1invite_users/post)
+
+
+## Switching Between Organizations
+
+If a user is invited to multiple organizations, they will have the ability to switch between them in the UI.
+User can click on the organization menu in the upper right corner, and choose one of the other available organizations
+from that menu to switch to it. If no other organizations appear, that user does not have access to any other
+organizations.
+

files/arthur-docs-markdown/platform-management/access-control/authenticating_with_multiple_organizations.md.txt

@@ -0,0 +1,3 @@
+# Access Control
+
+The access control documentation has moved. Please see the new guides in {doc}`../access-control-overview/index`

files/arthur-docs-markdown/platform-management/access-control/authentication.md.txt

@@ -0,0 +1,3 @@
+# Access Control
+
+The access control documentation has moved. Please see the new guides in {doc}`../access-control-overview/index`

files/arthur-docs-markdown/platform-management/access-control/authorization_custom.md.txt

@@ -0,0 +1,3 @@
+# Access Control
+
+The access control documentation has moved. Please see the new guides in {doc}`../access-control-overview/index`

files/arthur-docs-markdown/platform-management/access-control/authorization_standard.md.txt

@@ -0,0 +1,3 @@
+# Access Control
+
+The access control documentation has moved. Please see the new guides in {doc}`../access-control-overview/index`

files/arthur-docs-markdown/platform-management/access-control/index.md.txt

@@ -0,0 +1,19 @@
+---
+orphan: true
+---
+[//]: # (This folder was the old structure of the access control docs.
+The file structure was kept to maintain any customer bookmarks and point them to the new documentation.)
+
+# Access Control
+
+The access control documentation has moved. Please see the new guides in {doc}`../access-control-overview/index`
+
+```{toctree}
+:maxdepth: 2
+:hidden: true
+
+Authentication <authentication>
+Standard Authorization <authorization_standard>
+Custom Authorization <authorization_custom>
+Authenticating With Multiple Organizations <authenticating_with_multiple_organizations>
+```

files/arthur-docs-markdown/platform-management/index.md.txt

@@ -0,0 +1,33 @@
+(platform_management)=
+
+# Platform Management
+
+This section covers administration of the Arthur platform. Here you will find information on Arthur's access control,
+user management, installation, and maintenance. Arthur is offered as a hosted SaaS platform and on-prem solution,
+deployable to existing Kubernetes clusters, virtual machines, and bare metal servers.
+
+## {doc}`access-control-overview/index`
+
+This section shows how to configure the access and permissions for your various Arthur users within your organization.
+
+## {doc}`installation/index`
+
+This section shows how to install the necessary components for integrating Arthur into your platform.
+
+## {doc}`ongoing-maintenance/index`
+
+This section covers information for platform administrators responsible for ongoing maintenance of the Arthur platform.
+
+## {doc}`reference/index`
+
+This section contains references to configuration templates and permission sets for managing your Arthur usage.
+
+```{toctree}
+:maxdepth: 2
+:hidden:
+
+Access Control <access-control-overview/index>
+Installation <installation/index>
+Ongoing Maintenance <ongoing-maintenance/index>
+Reference <reference/index>
+```

files/arthur-docs-markdown/platform-management/installation/externalize_postgres.md.txt

@@ -0,0 +1,38 @@
+# Externalizing the Relational Database
+
+If desired, you can bring your own Postgres instance to use as your Arthur's relational database. Follow the steps 
+on this page to prepare your Postgres instance.
+
+First, deploy your Postgres instance in your desired environment with appropriate ingress firewall configuration.
+
+Create databases for the Arthur platform.
+```
+CREATE DATABASE arthurai
+CREATE DATABASE alert_service;
+CREATE DATABASE dataset_service;
+CREATE DATABASE metric_service;
+
+-- for stand alone instance
+CREATE USER arthurai WITH PASSWORD 'SuperSecret';
+-- for RDS instance
+CREATE ROLE arthurai WITH PASSWORD 'SuperSecret' LOGIN;
+
+REVOKE ALL PRIVILEGES ON DATABASE postgres FROM arthurai;
+GRANT ALL PRIVILEGES ON DATABASE arthurai TO arthurai;
+GRANT ALL PRIVILEGES ON DATABASE alert_service TO arthurai;
+GRANT ALL PRIVILEGES ON DATABASE dataset_service TO arthurai;
+GRANT ALL PRIVILEGES ON DATABASE metric_service TO arthurai;
+```
+
+If you have been using the embedded database and you wish to switch to using an external Postgres, backup the embedded 
+database and restore it to the new external Postgres with `pg_dump` and `pg_restore`.
+
+### Connecting to the database using SSL/TLS
+
+If your postgres instance supports SSL/TLS connections, and you want to connect to your external database
+with an encrypted connection, you simply need to set `Database SSL Mode` in the initial configuration. By default, this
+is set to `disable`. However, you can enable an encrypted connection using the value `require`.
+
+```{note}
+An externally managed Postgres instance is strongly recommended for production-grade installs.
+```

files/arthur-docs-markdown/platform-management/installation/high_availability.md.txt

@@ -0,0 +1,54 @@
+# Setting up for High Availability
+
+## Introduction
+
+The Arthur Platform is built to run in a High Availability configuration, ensuring that the application can function
+in the event of a Data Center outage, hardware outages, or other similar infrastructure issues.
+
+In order to take advantage of this, there are a few requirements in how your infrastructure is setup:
+
+1. Installing across 3 Availability Zones
+2. Specifying the correct Instance Types
+3. Configuring the cluster for Auto-scaling
+
+## Notes about this document
+
+```{note}
+Note that this document is written using AWS terminology, as this is one of the environments/infrastructure that Arthur uses
+internally for our environments. However, these setup steps should work across various cloud providers using similar features.
+```
+
+```{note}
+Note that this document is written with the pre-requisite that you are installing Arthur in a High Availability configuration.
+At the minimum, this means that there should be 3 instances across which Arthur is deployed.
+```
+
+## Installing across 3 Availability Zones
+
+In order to ensure continuous operation during an Availability Zone (AZ) outage, Arthur must be installed on a cluster
+that has 3 Availability Zones. This ensures that in the event of one AZ outage that the rest of the components can still
+operate.
+
+To do this in AWS, create 3 separate Auto-Scaling Groups (ASGs) - one for each AZ. You can configure which AZ an ASG provisions
+instances into when you create the ASG.
+
+When Arthur deploys, the stateful services (eg: databases, messaging queues, etc.) will be balanced across the 3 AZs automatically using
+kubernetes pod anti-affinity rules (pods will not schedule onto nodes where there already exists another pod that is of the same component).
+
+## Specifying the correct Instance Types
+
+Generally speaking, the best way to ensure you have deployed the correct Instance Types is to monitor resource utilization across the cluster
+to determine when your services are hitting resource limits.
+
+When initially configuring a cluster for Arthur, we recommend 3 nodes, where each node has at least 16 vCPUs and 64G of RAM (eg: an `m5a.4xlarge` instance type).
+This is a good starting point for a general-purpose cluster that will grow with your production usage.
+
+## Configuring the cluster for Auto-scaling
+
+Arthur's stateless components horizontally auto-scale, but in order to take the maximum advantage of this, you will need to configure and install
+an additional component that performs node autoscaling (eg: adding more instances).
+
+AWS specifies how to setup cluster autoscaling in their [documentation](https://docs.aws.amazon.com/eks/latest/userguide/autoscaling.html).
+
+Generally speaking, it involves setting up an IAM role and granting permissions to autoscale the cluster, and then installing a third-party component
+to perform the autoscaling (eg: [cluster-autoscaler](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md)

files/arthur-docs-markdown/platform-management/installation/index.md.txt

@@ -0,0 +1,34 @@
+# Installation
+
+Arthur can be installed to existing Kubernetes clusters (K8s) or virtual machines (VM). For both Kubernetes and virtual machine deployments, Arthur supports two types of installation methods, online and airgapped. The online method allows fast and continuous differential upgrades. The airgapped install supports environments where the cluster has no outbound internet connectivity.
+
+## {doc}`Installation Requirements <requirements>`
+This page covers the requirements needed in your system to deploying Arthur within your system.
+
+## {doc}`Platform Readiness FAQ <platform_readiness_faq>`
+This page covers some of the most frequently-asked questions about setting up Arthur for on-premises installation.
+
+## {doc}`Configuring for High Availability <high_availability>`
+This page covers how to configure your infrastructure to support running Arthur as a High Availability Application.
+
+## {doc}`kubernetes-installation/index`
+This section shows how to install Arthur into a Kubernetes cluster.
+
+## {doc}`vm-installation/index`
+This section shows how to install Arthur into a virtual machine.
+
+## {doc}`Externalize Postgres <externalize_postgres>`
+This page shows how to set up a Postgres instance to use as your Arthur's relational database.
+
+
+```{toctree}
+:maxdepth: 2
+:hidden:
+
+Installation Requirements <requirements>
+Platform Readiness <platform_readiness_faq>
+Kubernetes Installation <kubernetes-installation/index>
+Virtual Machine Installation <vm-installation/index>
+Configuring for High Availability <high_availability>
+Externalize Postgres <externalize_postgres>
+```

files/arthur-docs-markdown/platform-management/installation/kubernetes-installation/index.md.txt

@@ -0,0 +1,31 @@
+# Kubernetes Installation
+
+This section covers the steps required for installing Arthur on a Kubernetes cluster. There are separate steps required for online and airgapped installations. An additional set of instructions are also available for installing Arthur that’s scoped within a K8s namespace.
+
+## {doc}`kubernetes-preparation/index`
+This section shows how to configure a Kubernetes cluster to install Arthur.
+
+## {doc}`k8s_install_online`
+This page shows how to install Arthur into an online Kubernetes installation.
+
+## {doc}`k8s_install_airgapped`
+This page shows how to install Arthur into an airgapped Kubernetes installation.
+
+## {doc}`k8s_install_airgapped_cli`
+This page shows how to install Arthur into an online Kubernetes installation via a command-line interface.
+
+## {doc}`k8s_install_namespace_scoped`
+This page shows how to install Arthur into a Kubernetes cluster with namespace scoped privileges.
+
+
+```{toctree}
+:maxdepth: 2
+:hidden:
+
+Configuring Kubernetes <kubernetes-preparation/index>
+Online <k8s_install_online>
+Airgapped <k8s_install_airgapped>
+Airgapped CLI <k8s_install_airgapped_cli>
+Namespace Scoped <k8s_install_namespace_scoped>
+
+```

files/arthur-docs-markdown/platform-management/installation/kubernetes-installation/k8s_install_airgapped.md.txt

@@ -0,0 +1,166 @@
+# Airgap Kubernetes Cluster (K8s) Install
+
+Make sure your K8s cluster is ready for Arthur platform installation by following
+{doc}`the K8s preparation guide <kubernetes-preparation/index>`.
+
+### **Preparing Container Registries**
+
+Prepare your private container image registry for Arthur artifacts by creating the following list of repositories:
+
+Admin Console:
+```
+arthurai/dex
+arthurai/kotsadm
+arthurai/kotsadm-migrations
+arthurai/local-volume-fileserver
+arthurai/local-volume-provider
+arthurai/minio
+arthurai/postgres
+```
+
+Application:
+```
+arthurai/alert-service
+arthurai/alpine
+arthurai/api-service
+arthurai/argocli
+arthurai/argoexec
+arthurai/aws-cli
+arthurai/beta-client
+arthurai/busybox
+arthurai/clickhouse-operator
+arthurai/clickhouse-server
+arthurai/client
+arthurai/cp-kafka
+arthurai/cp-kafka-connect
+arthurai/cp-schema-registry
+arthurai/cp-zookeeper
+arthurai/custom-hpa
+arthurai/dataset-service
+arthurai/ingestion-service
+arthurai/kafka-connect-monitor
+arthurai/kafka-exporter
+arthurai/kafka-prometheus-jmx-exporter
+arthurai/kubectl
+arthurai/mc
+arthurai/metric-service
+arthurai/metrics-exporter
+arthurai/minio
+arthurai/model-server
+arthurai/model-server-controller
+arthurai/postgresql
+arthurai/python-jobs
+arthurai/python-spark-jobs
+arthurai/pytorch-jobs
+arthurai/query-service
+arthurai/redis
+arthurai/scala-spark-jobs
+arthurai/schema-service
+arthurai/workflow-controller
+arthurai/zookeeper-exporter
+```
+
+As an example, here's how you can create a new `arthurai/alert-service` repository on AWS ECR.
+```shell
+export AWS_REGION=<your_region>
+aws ecr create-repository --repository-name=arthurai/alert-service
+```
+
+### **Download Installation Files**
+
+Go to the download portal using the URL and the password provided by Arthur.
+
+Select the "Bring my own cluster" option.
+![.png image](/_static/images/platform/download_portal_airgap_existingk8s.png)
+
+Click the “Download license” button to download your license in YAML file.
+
+Download the "KOTS Airgap Bundle" and the "arthur Airgap Bundle".
+
+### **Setup for Installation**
+
+Make sure you're in the correct kubectl environment context before running the installer.
+```shell
+kubectl config current-context
+```
+
+Install the KOTS kubectl extension on your local machine:
+```shell
+curl https://kots.io/install | bash
+```
+
+If the Linux workstation you're running `kubectl` from is also in the airgap environment, download the "KOTS CLI" from the download portal and install it like below:
+```shell
+tar zxf kots_linux_amd64.tar.gz
+# move it to a location that's on your path
+sudo mv kots /usr/local/bin/kubectl-kots
+```
+
+```{note}
+The "KOTS CLI" and "KOTS Airgap Bundle" must be installed at the same time and therefore will be on the same version.
+```
+```shell
+kubectl kots version
+```
+
+If your workstation is a Mac, you can download the latest version of Kots CLI Darwin binary from [https://kots.io/](https://kots.io/).
+
+### **Start Installation**
+
+Push the Admin Console images to your private registry:
+```shell
+kubectl kots admin-console push-images ./kotsadm.tar.gz [Your private registry host]/arthurai \
+  --registry-username [Read-Write Username] \
+  --registry-password [Read-Write Password]
+```
+
+As an option, you can also pre-upload the application images to your private registry before running the installer:
+```shell
+kubectl kots admin-console push-images ./arthur-x.x.x.airgap [Your private registry host]/arthurai \
+  --registry-username [Read-Write Username] \
+  --registry-password [Read-Write Password]
+```
+
+Install the Admin Console (see here for {doc}`Namespace-Scoped Installs <k8s_install_namespace_scoped>`):
+```shell
+kubectl kots install arthur \
+  --no-port-forward \
+  --namespace arthur \
+  --shared-password [Provide an Admin Console password] \
+  --kotsadm-namespace arthurai \
+  --kotsadm-registry [Your private container image repository] \
+  --registry-username [Read-Write Username] \
+  --registry-password [Read-Write Password]
+```
+
+Create a port forwarding tunnel to Admin Console. Go to `http://localhost:8800` to access the Admin Console:
+```shell
+kubectl kots admin-console --namespace arthur
+```
+
+Follow the instructions on the Admin Console to complete your installation by providing the private registry details and `arthur-x.x.x.airgap` bundle.
+![.png image](/_static/images/platform/install-airgap.png)
+
+```{note}
+The upload process can take couple of hours so ensure your laptop does not go to sleep. You may follow the instructions {doc}`here <k8s_install_airgapped_cli>` to install the Admin Console and Arthur app programmatically using the CLI only.
+```
+
+Configure Arthur.
+![.png image](/_static/images/platform/configure.png)
+
+### **Verify Installation**
+
+Monitor the Admin Console dashboard for the application status to become **Ready**.
+
+![.png image](/_static/images/platform/dashboard.png)
+
+To see the progress of the deployment, monitor the deployment status with `kubectl` CLI:
+```shell
+kubectl get deployment,statefulset,pod -n arthur
+```
+
+If anything is showing Pending, it is likely you need to add more/bigger nodes to your cluster.
+
+### **Customize Installation**
+
+Configure graphs on Admin Console by clicking on the `Configure Prometheus Address` button and providing your Prometheus endpoint (e.g. `http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090`).

files/arthur-docs-markdown/platform-management/installation/kubernetes-installation/k8s_install_airgapped_cli.md.txt

@@ -0,0 +1,35 @@
+# Airgap Kubernetes Cluster (K8s) Install with CLI
+
+If you prefer to install programmatically using CLI only, follow the steps below.
+
+Prepare a `config.yaml` file using {doc}`the configuration template </platform-management/reference/config_template>`.
+
+Deploy the application by running the below `kubectl` command:
+```shell
+kubectl kots install arthur \
+  --no-port-forward \
+  --namespace arthur \
+  --shared-password [Provide an Admin Console password] \
+  --license-file ./license.yaml \
+  --config-values ./config.yaml \
+  --airgap-bundle ./arthur-x.x.x.airgap \
+  --kotsadm-registry [Your private container image repository] \
+  --kotsadm-namespace arthurai \
+  --registry-username [Read-Write Username] \
+  --registry-password [Read-Write Password]
+```
+
+`shared-password` is the Admin Console password.
+
+
+## Installing a specific version of Arthur
+
+To install a specific version of Arthur, you would run the same command as above (following the same steps to prepare the configuration), with the inclusion of the
+`--app-version-label` flag. This flag allows you to specify which specific version of Arthur you want to install (eg: to setup a sandbox environment on the same version
+as production).
+
+To determine which versions of Arthur are available, you can run:
+
+```
+kubectl kots get versions arthur -n <arthur namespace>
+```

files/arthur-docs-markdown/platform-management/installation/kubernetes-installation/k8s_install_namespace_scoped.md.txt

@@ -0,0 +1,71 @@
+# Kubernetes Cluster (K8s) Install with Namespace Scope Privileges
+
+If you would like to install the Arthur platform with namespace scoped privileges, there are certain components that will fail since they will need cluster level access. 
+These cluster-level components are CRDs ([Custom Resource Definitions](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/)) which are _**required**_ for proper functioning and operation of the Arthur platform. However, these CRDs only need to be installed once with cluster admin privileges, and elevated access is not required for normal usage of the platform. 
+
+```{note} Both the Admin Console and Arthur application can be installed at the cluster-level or namespace-scope independent of each other.
+```
+
+### CRDs leverage by Arthur Platform
+
+The Arthur platform makes use of the following two CRDs:
+1. **Argo Workflows** : Kubernetes-native Open-source workflow manager
+1. **ClickHouse Operator** : Column-oriented OLAP datastore
+
+## Installing Admin Console within a Namespace
+
+By default, the Admin Console is installed at the Cluster level, available in all namespaces. If you would like to install the Admin Console only within a specific namespace, you can use the following flag to the `kots` command:
+```shell
+kubectl kots install arthur \
+  --use-minimal-rbac
+  --skip-rbac-check 
+```
+
+Since the Admin Console will not have access to the Cluster, certain Preflight checks will fail. It is the responsibility of the Cluster Admin to ensure sufficient resources are provisioned with the correct version of K8s.
+
+## Installing Cluster-level CRDs for Arthur, from Nexus
+
+Since the Arthur platform requires CRDs for normal operation, these will need to be installed by the Cluster Admin prior to installing Arthur itself, in no particular order. The instructions below show how you can download the CRD charts from our publicly hosted repository.
+
+* Argo Workflows:
+```shell
+helm repo add arthurai-released https://repository.arthur.ai/repository/charts --username <nexus-username> --password <nexus-password>
+helm install argo-workflows-crd arthurai-released/argo-workflows-crd --version 0.19.1-arthur-1
+```
+
+* ClickHouse Operator:
+```shell
+helm repo add arthurai-released https://repository.arthur.ai/repository/charts --username <nexus-username> --password <nexus-password>
+helm install clickhouse-operator-crd arthurai-released/clickhouse-operator-crd --version 0.19.2-arthur-1
+```
+
+```{note} Please reach out to our Sales Team if you do not have credentials to our Nexus repository.
+```
+
+## Installing Cluster-level CRDs for Arthur, from Airgap Bundle
+
+If you are in an airgapped environment with no access to the public internet, the CRD charts are also available in the Airgap bundle provided to you. The instructions below show how you can extract the charts from the Airgap bundle.
+
+```shell
+tar -xvf arthur-<version>.airgap
+cd arthur-<version>
+tar -xvf app.tar.gz
+cd app
+```
+
+* Argo Workflows:
+```shell
+helm install argo-workflows-crd argo-workflows-crd-0.14.0-arthur-2.tgz
+```
+
+* ClickHouse Operator:
+```shell
+helm install clickhouse-operator-crd arthurai-released/clickhouse-operator-crd-0.18.4-arthur-2.tgz
+```
+
+You can verify the CRDs have been installed successfully, by executing the following command:
+```shell
+kubectl get crd | grep -iE 'argo|clickhouse'
+```
+
+Now that we have the pre-reqs installed with elevated access, we can now switch over to namespace-scoped access to complete the installation either {doc}`using the Admin Console <k8s_install_airgapped>` or {doc}`using the CLI <k8s_install_airgapped_cli>`.

files/arthur-docs-markdown/platform-management/installation/kubernetes-installation/k8s_install_online.md.txt

@@ -0,0 +1,59 @@
+# Online Kubernetes Cluster (K8s) Install
+
+Make sure your K8s cluster is ready for Arthur platform installation by following {doc}`the K8s preparation guide <kubernetes-preparation/index>`.
+
+### **Download Installation Files**
+
+Go to the download portal using the URL and the password provided by Arthur.
+
+Click the "Download license" button to download your license in YAML file.
+
+### **Setup for Installation**
+
+Make sure you're in the correct kubectl environment context before running the installer.
+
+Install the KOTS kubectl extension on your local machine:
+```shell
+curl https://kots.io/install | bash
+```
+
+### **Start Installation**
+
+Run the Admin Console installer and login on your browser at `localhost:8800` via the provided port forwarding tunnel:
+```shell
+kubectl kots install arthur
+```
+For Namespace-Scoped Installs, follow this {doc}`guide <k8s_install_namespace_scoped>`.
+
+When you need to re-create the tunnel to Admin Console, run:
+```shell
+kubectl kots admin-console --namespace <your_name_space>
+```
+
+Upload your license file.
+![.png image](/_static/images/platform/license.png)
+
+On the following screen, click on the link to install Arthur from the Internet.
+![.png image](/_static/images/platform/install-online.png)
+
+Configure Arthur.
+![.png image](/_static/images/platform/configure.png)
+
+Review the preflight checks to make sure that your machine meets the minimum requirements before you proceed with the installation.
+
+### **Verify Installation**
+
+Monitor the admin console dashboard for the application status to become **Ready**.
+
+![.png image](/_static/images/platform/dashboard.png)
+
+To see the progress of the deployment, monitor the deployment status with `kubectl` CLI:
+```shell
+kubectl get deployment,statefulset,pod -n <yournamespace>
+```
+
+If anything is showing Pending, it is likely you need to add more/bigger nodes to your cluster.
+
+### **Customize Installation**
+
+Configure graphs on Admin Console by clicking on the `Configure graphs` button and providing your Prometheus endpoint (e.g. `http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090`).

files/arthur-docs-markdown/platform-management/installation/kubernetes-installation/kubernetes-preparation/index.md.txt

@@ -0,0 +1,19 @@
+# Kubernetes Installation Preparation
+
+This section covers the steps required to prepare an existing Kubernetes cluster for installing the Arthur platform.
+
+## {doc}`k8s_install_prep`
+This page shows the basic process of configuring an existing K8s cluster for installing the Arthur platform.
+
+## {doc}`k8s_install_prep_aws`
+This page shows the additional steps involved in configuring an Amazon AWS EKS cluster for installing the Arthur platform.
+
+
+```{toctree}
+:maxdepth: 1
+:hidden:
+
+Configuring Kubernetes <k8s_install_prep>
+Additional Configurations for Amazon AWS <k8s_install_prep_aws>
+
+```

files/arthur-docs-markdown/platform-management/installation/kubernetes-installation/kubernetes-preparation/k8s_install_prep.md.txt

@@ -0,0 +1,135 @@
+# Kubernetes Cluster (K8s) Install Preparation
+
+This is a guide to help you prepare your existing Kubernetes cluster for installing the Arthur platform.
+The examples use Helm 3.
+
+Make sure you're in the correct `kubectl` environment context before running the installer.
+
+## **Install Prometheus**
+
+Example:
+```shell
+helm repo add \
+  prometheus-community \
+  https://prometheus-community.github.io/helm-charts
+helm repo update
+helm upgrade --install -n monitoring \
+  --create-namespace \
+  kube-prometheus-stack \
+  prometheus-community/kube-prometheus-stack \
+  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false
+helm upgrade --install -n monitoring \
+  --create-namespace \
+  prometheus-adapter \
+  prometheus-community/prometheus-adapter
+```
+
+Verify that Prometheus CRDs are installed:
+```shell
+kubectl api-resources | grep monitoring
+```
+
+Verify that Prometheus is up and running:
+```shell
+kubectl --namespace monitoring get pods -l "release=kube-prometheus-stack"
+```
+
+If everything is installed correctly, the following command should not return _"ServiceUnavailable"_:
+```shell
+kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1
+```
+
+(k8s_install_prep_install_ingress)=
+## **Install Ingress**
+
+Example with Nginx:
+```shell
+helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
+helm repo update
+helm upgrade --install -n ingress-system \
+  --create-namespace \
+  ingress-nginx \
+  ingress-nginx/ingress-nginx
+```
+
+[Optional] To add an AWS managed SSL certificate, create a `values.yaml` file with following contents -
+```shell
+controller:
+  service:
+    annotations:
+      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
+      service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
+      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
+      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <AWS managed SSL certificate ARN>
+      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
+      service.beta.kubernetes.io/aws-load-balancer-type: elb
+      service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: ELBSecurityPolicy-TLS-1-2-2017-01
+      service.beta.kubernetes.io/aws-load-balancer-internal: true # optional annotation that creates a non-internet-facing loadbalancer
+    targetPorts:
+      http: "tohttps"
+
+  allowSnippetAnnotations: "true"
+  config:
+    http-snippet: |
+      server {
+        listen 2443;
+        return 308 https://$host$request_uri;
+      }
+    use-forwarded-headers: "true"
+
+  containerPort:
+    http: 8080
+    tohttps: 2443
+    https: 80
+```
+Upgrade or install the helm chart with the `values.yaml` you created.
+```shell
+helm upgrade --install -n ingress-system \
+  --create-namespace \
+  ingress-nginx \
+  ingress-nginx/ingress-nginx \
+  -f values.yaml
+```
+
+If you need to install Nginx in the same namespace as Arthur (not recommended) and want to use our network-policy to restrict ingress to the Arthur application, use the below command to add labels to the pods and services.  The network-policy allows traffic between pods and services that have these labels.
+```shell
+helm upgrade --install -n arthur --set controller.podLabels.network-app=arthurai,controller.service.labels.network-app=arthurai,defaultBackend.podLabels.network-app=arthurai,.service.labels.network-app=arthurai \
+  ingress-nginx \
+  ingress-nginx/ingress-nginx
+```
+
+Look up the hostname for the Ingress and configure it in your DNS (e.g. `arthur.mydomain.com`).
+```shell
+kubectl get svc -n ingress-system ingress-nginx-controller -ojsonpath='{.status.loadBalancer.ingress[*].hostname}'
+```
+
+## **Install Metrics Server**
+
+Example:
+```shell
+helm repo add bitnami https://charts.bitnami.com/bitnami
+helm repo update
+helm upgrade --install -n monitoring \
+  --create-namespace \
+  metrics-server \
+  bitnami/metrics-server \
+  --set apiService.create=true \
+  --set --extraArgs.kubelet-preferred-address-types=InternalIP
+```
+
+Verify that you can retrieve metric snapshots.
+```shell
+kubectl top node
+```
+
+## **Configure the cluster-autoscaler**
+
+In a production environment, it is vital to ensure that there are enough resources (memory and cpu) available for pods to get scheduled on the Kubernetes cluster. Please follow the instructions for your cloud provider to install the [cluster-autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler) on your cluster.
+
+Verify that the `cluster-autoscaler` is successfully installed.
+```shell
+kubectl get deployments -n kube-system | grep -i cluster-autoscaler
+```
+
+## **Cloud Provider-specific Configuration**
+If installing on an existing Amazon AWS EKS, follow the additional steps {doc}`here <k8s_install_prep_aws>`.

files/arthur-docs-markdown/platform-management/installation/kubernetes-installation/kubernetes-preparation/k8s_install_prep_aws.md.txt

@@ -0,0 +1,126 @@
+# Deploying on Amazon AWS EKS
+
+This is a guide with additional steps to help you prepare your existing Amazon Elastic Kubernetes Service (Amazon EKS) cluster for installing the Arthur platform.   
+Ensure the initial steps detailed {doc}`here <k8s_install_prep>` have already been applied to the cluster.
+
+## **Configure EKS EBS CSI driver**
+
+As of EKS 1.23, the Amazon Elastic Block Store (Amazon EBS) Container Storage Interface (CSI) driver needs to be installed explicitly. This driver allows EKS clusters to manage the lifecycle of EBS volumes for Persistent Volumes. For more information, see [Amazon Docs](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html).
+
+If you are deploying Arthur on EKS 1.23+, you will need to follow the instructions on [this page](https://docs.aws.amazon.com/eks/latest/userguide/managing-ebs-csi.html).
+
+Verify that the Add-On is successfully installed, by navigating to **AWS Console &rarr; EKS &rarr; Cluster &rarr; Add-Ons** or by running `helm list -A`, depending on your installation method.
+
+## **Optimizing the AWS EKS StorageClass**
+Once the EKS EBS CSI driver is installed, you can take advantage of the `gp3` StorageClass type. This StorageClass is more [cost-effective and performant](https://aws.amazon.com/blogs/storage/migrate-your-amazon-ebs-volumes-from-gp2-to-gp3-and-save-up-to-20-on-costs/)  than the previous `gp2` StorageClass. Apply the below YAML definition to your cluster:
+```yaml
+    apiVersion: storage.k8s.io/v1
+    kind: StorageClass
+    metadata:
+      annotations:
+        storageclass.kubernetes.io/is-default-class: "true"
+      name: gp3
+    parameters:
+      type: gp3
+    provisioner: ebs.csi.aws.com
+    reclaimPolicy: Delete
+    volumeBindingMode: WaitForFirstConsumer
+    allowVolumeExpansion: true
+```
+```{note}
+Ensure there is **_only one_** default StorageClass on the cluster. This is controlled by the `storageclass.kubernetes.io/is-default-class` annotation.
+```
+
+## **Supported AWS Service Authentication Mechanisms**
+If using AWS services with Arthur such as S3 or SES, you will need to configure Arthur to authenticate with AWS.  Arthur currently supports 3 authentication mechanisms:
+
+![iam auth](/_static/images/platform/iam_authentication.png)
+
+### AWS Access Keys
+Access Keys only work with S3.  If you want to use Access Keys you will need to provision an IAM user and a set of keys.  Via [AWS IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_change-permissions.html), you will need to grant this user read/write access to the S3 storage bucket that you plan on using with Arthur.  Selecting the Access Keys option will expand the Blob Storage section of the config page where you will be able to enter your Access key, Secret Access key ID, and the S3 bucket.
+
+![s3](/_static/images/platform/s3_access_keys.png)
+
+## IRSA
+
+We recommend using [IRSA](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) to authenticate Arthur with AWS as it is the most secure, and is the only mechanism that supports SES.  Using this methodology will require a bit of AWS platform work in preparation for Arthur.  You can follow these [AWS docs](https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html) which will show you how to do this setup via eksctl or the AWS CLI, or you can automate this via your internal Infrastructure as Code.  
+
+The role that you create will need to have S3 read/write privileges on the bucket you want to use with Arthur, and permissions to send email via your SES entity.  Example snippets are as below:
+
+### Sample IAM policy for S3 access:
+```
+{
+    "Statement": [
+        {
+            "Action": [
+                "s3:PutObject",
+                "s3:GetObject",
+                ...
+            ],
+            "Effect": "Allow",
+            "Resource": [
+                "arn:aws:s3:::<insert-s3-bucket-name>/*",
+                "arn:aws:s3:::<insert-s3-bucket-name>"
+            ],
+            ....
+        },
+```
+
+### Sample IAM policy for SES access:
+```
+            "Action": [
+                "ses:SendTemplatedEmail",
+                "ses:SendEmail",
+                "ses:SendCustomVerificationEmail",
+                "ses:SendBulkTemplatedEmail",
+                "ses:SendBulkEmail",
+                "ses:SendBounce"
+            ],
+            "Effect": "Allow",
+            "Resource": "*",
+            "Sid": "sesSendEmails"
+        },
+```
+
+This role will also need to have a trust relationship to the OIDC provider of your EKS cluster, specifying the Arthur service accounts.  See the linked docs above for a further explanation.  An example snippet of this is:
+```
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Sid": "",
+            "Effect": "Allow",
+            "Principal": {
+                "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.us-east-2.amazonaws.com/id/ABDCEF......"
+            },
+            "Action": "sts:AssumeRoleWithWebIdentity",
+            "Condition": {
+                "StringEquals": {
+                    "oidc.eks.us-east-2.amazonaws.com/id/ABDCEF:sub": [
+                        "system:serviceaccount:<namespace>:arthurai-<namespace>",
+                        "system:serviceaccount:<namespace>:arthurai-<namespace>-helm-hook"
+                    ]
+                }
+            }
+        }
+    ]
+}
+```
+
+Once this is all set up you can pass this role into Arthur via the config page.  This sets the role in the Arthur Service Accounts specified above, which enables Arthur's pods to authenticate with AWS via the role, and the permissions you created.  Be sure to use the exact formatting shown below:
+
+![irsa](/_static/images/platform/irsa.png)
+
+Proceed to the Blob Storage section of the Arthur config page to specify the S3 bucket 
+
+### SES
+
+To utilize AWS SES for Arthur generated emails, you will need to configure IRSA as outlined in the above section.  Once this is done navigate to the email configuration section of Arthur's config page.  Select AWS SES, and then enter the region in which your SES entity is configured. As outlined above, the role associated with the cluster must have permissions on this SES entity.  If the SES entity is in the same account as your cluster, and you do not need to utilize a different role such as for cross account permissions, do not enter a role in the second box.  
+
+![ses](/_static/images/platform/ses.png)
+
+If your SES entity is in another Arthur account you will need to set up cross account privileges between roles.  In the account of your SES entity (Account A) you will need to create an IAM role (Role A) that has send email permissions to SES as depicted above.   Role A will also need to have a trust relationship with either the account that your cluster is in (Account B), the OIDC provider on your cluster as depicted above, or the IRSA role associated with your cluster.  Additionally, the IRSA role that you created above in Account B, will also need to be granted STS assume role privileges on the role you are creating in Account A. 
+
+Once all of this is set up, enter the role in the account that contains the SES entity (Account A), that the IRSA role should assume to send emails:  
+
+![ses cross account](/_static/images/platform/ses_cross_account.png)

files/arthur-docs-markdown/platform-management/installation/platform_readiness_faq.md.txt

@@ -0,0 +1,96 @@
+# Platform Readiness for Existing Cluster Installs
+
+The Arthur platform can be installed on an on-prem or cloud-based pre-provisioned Kubernetes cluster, so all the data and controls adhere to existing corporate practices.
+
+## Arthur Cluster Installs FAQs
+
+### SysAdmin
+
+1. **What kind of privileges/hardware does the SysAdmin installing the Arthur platform need?**   
+The SysAdmin will need a workstation with the following requirements:
+   - running Linux or MacOS. The KOTS CLI installer does not support Windows.
+   - sudo/root access. To install the KOTS CLI plugin for `kubectl`.
+   - connection to the Kubernetes cluster, using `kubectl`, and privileges to deploy K8s Objects at either Cluster or Namespace scope (at least).
+   - (recommended) access to the Internet. For downloading the installer, plugins and fetching updates.
+
+1. **How can I download the artifacts required for installing the Arthur platform?**   
+All artifacts required for installing the Arthur platform are available on a customer-specific password-protected portal, which your sales team can give you access to. It is  recommended that the portal is accessible from within your corporate network, since the artifacts are around mutiple GBs in size.
+
+1. **Does my kubernetes cluster need access to the internet?**   
+The Arthur platform can be installed without Internet access, once all the required files are downloaded and available locally. However, we recommend access to the Internet from the Kubernetes cluster for an efficient install and upgrade experience. Please inform your sales team about any network restrictions and optionally, if its possible to {ref}`whitelist specific URLs <requirements_for_online_installation>`. 
+
+### Cloud Providers
+
+1. **Which Kubernetes distributions that Arthur supports out-of-the-box?**   
+Arthur is architected to run on any distribution of Kubernetes, however certain commercial distributions are untested. The Arthur application is validated/tested on:
+   - Amazon [AWS EKS](https://aws.amazon.com/eks/)
+   - Microsoft [Azure AKS](https://azure.microsoft.com/en-us/services/kubernetes-service/)
+
+1. **Which cloud providers has Arthur been tested on?**   
+The Arthur platform has been tested on the following cloud providers:
+   - Amazon [AWS](https://aws.amazon.com/)
+   - Microsoft [Azure](https://azure.microsoft.com/en-us/)  
+
+1. **What container runtimes does Arthur support?**   
+Containers in the Arthur Platform run on the following container runtimes:
+   - docker (slated to be [deprecated in Kubernetes 1.24](https://kubernetes.io/blog/2022/01/07/kubernetes-is-moving-on-from-dockershim/))
+   - containerd
+
+### Kubernetes Server
+
+1. **What version(s) of Kubernetes Server does Arthur support?**   
+Arthur supports Kubernetes Server 1.19 through 1.22.
+
+1. **Can the Arthur platform be scoped to a dedicated namespace?**   
+The Arthur platform can be deployed and scoped to a specific namespace, though there are some cluster-level CustomResourceDefinitions that need to be pre-installed. See details [here](./kubernetes-installation/k8s_install_namespace_scoped).
+
+1. **What are the minimum resource requirements for operating the Arthur Platform?**   
+Optimal performance of the Arthur platform is ensured on a 6 node cluster (though test clusters can be provisioned with 3 nodes) with each node having 16 CPUs, 32GB Memory (RAM) and 1000 GB Storage with atleast 3000 IOPS. However, please reach out to your sales team for a tailored configuration custom to your projected workloads.
+
+1. **Is there a default StorageClass defined on the Kubernetes cluster?**   
+The Kubernetes cluster must have a default StorageClass defined before starting the Arthur platform installation. If a default StorageClass does not exist, adding the `storageclass.kubernetes.io/is-default-class: "true"` annotation to a StorageClass should remedy this requirement.
+
+1. **What Ingress Controller are you planning to use to access the Arthur platform? Is it already installed?**   
+The Arthur platform needs to expose a couple of services so the application is accessible outside the cluster. All [compatible Kubernetes Ingress controllers](https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/) should work, though {ref}`Nginx Ingress Controller <k8s_install_prep_install_ingress>` installed in a separate namespace is recommended.
+
+1. **Are there any SecurityContext requirements on the Kubernetes cluster?**   
+The Arthur platform is architected to leverage as few permissions as deemed necessary for optimal functioning. No container is run as root. All processes are owned by non-system users. Please reach out to your sales team if you have specific SecurityContext requirements.
+
+1. **Does Arthur support running on SELinux environments?**   
+The Arthur platform requires SELinux to be running in permissive mode, if enabled.
+
+1. **Are there any Network Policies configured on the Kubernetes cluster?**   
+Pods of the Arthur platform will need to communicate with each other. By default, pods [can communicate](https://kubernetes.io/docs/concepts/services-networking/network-policies/#default-policies) with each other. Please reach out to your sales team if you have custom Network Policies configured on the Kubernetes cluster.
+
+1. **How many IP addresses should be available for the Arthur Platform?**   
+The Arthur platform is architected to be scalable, using resources on-demand. Given the dynamic nature of the infrastructure involved, we recommend at least 128 IP address CIDR blocks attached to the relevant subnets. However, this number can increase as more models are onboarded to the platform.
+
+1. **Are there any namespace-level constraints enforced on the Kubernetes cluster?**   
+Please let your sales team know if there are any constraints configured at the namespace-level on the Kubernetes cluster, as this will help prepare for a smooth installation experience.
+
+1. **Are there any cluster-level constraints enforced on the Kubernetes cluster?**   
+Please let your sales team know if there are any specific cluster-level contraints configured on the Kubernetes cluster, as this will help prepare for a smooth installation experience.
+
+1. **Does the Kubernetes cluster have access to a private/public container registry?**   
+The Kubernetes cluster on which the Arthur platform will be installed must have connectivity to a container registry. The SysAdmin performing the installation must also have Read/Write access to the same container registry.
+
+1. **Does the Kubernetes cluster have access to a private/public Pypi/Conda registry?**   
+The Kubernetes cluster on which the Arthur platform will be installed must have connectivity to a Pypi/Conda registry, which ensures optimum utilization of the features of the platform.
+
+### Other Considerations
+
+1. **Does your enterprise have a software procurement process?**   
+Please keep your sales team informed of any software procurement process that maybe in place before installing new software, and potential turnaround times for such processes.
+
+1. **Do you want to deploy Arthur on infrastructure that isn't mentioned above (eg: Cloud Providers, Kubernetes Distributions, etc.)?**  
+If so, please inform your sales team as soon as possible so we can setup an architecture review between your platform team and Arthur's platform team. 
+
+1. **Can any of the Arthur platform components be externalized, so its not managed by Arthur?**   
+The platform supports the use of AWS S3 as well as most S3 compatible systems as the Object/Blob store. The embedded metadata database can be replaced by a recent version of Postgres.   
+A managed service for S3 and/or Postgres is recommended for production-grade installs.
+
+1. **Can the Arthur platform be deployed on a Kubernetes cluster that is shared with other applications?**  
+The Arthur platform has been architected to be highly scalable and reliable. Based on usage (number of models) and load (data ingestion), pods are scaled in short periods of time to ensure efficient operation. As such, if other applications will be installed on the same Kubernetes platform, talk to your sales team about provisioning dedicated nodegroups for the cluster.
+
+3. **Does the Arthur platform support different organizations/business units using the same application?**  
+Yes. See our guide on [User and Org Management](../ongoing-maintenance/organizations_and_users). 

files/arthur-docs-markdown/platform-management/installation/requirements.md.txt

@@ -0,0 +1,84 @@
+# On-prem Deployment Requirements
+
+## General
+* A DNS hostname
+* TLS private key & certificate
+* SMTP server (StartTLS supported)
+
+The minimum compute resource requirements in this documentation is for running a few small models in a non-production environment. Your production deployment will likely use more compute resources to achieve higher availability, performance and scalability. 
+
+Arthur’s horizontally elastic architecture allows high throughput processing in both streaming and batch. The platform's auto-scaler mechanism self-manages resource utilization in optimized and cost-effective fashion. It automatically scales up and down based on compute resource requests by the platform activities as well as the lag observed in the data pipeline queue within the limits of the allocated hardware. This works best in a cloud infrastructure with a managed Kubernetes service that enables Arthur to also auto-scale the provisioned hardware (e.g. AWS EKS, Azure ASK).
+
+Storage volumes used for Arthur deployment should be encrypted with a data key using industry-standard data encryption (e.g. AES-256). This applies to the mounted disk volumes as well as the externalized storage, such as the S3 object storage and the relational database if any.
+
+## Kubernetes Install
+* Kubectl-ing workstation: Linux or MacOS 
+* Kubernetes: 1.22 to 1.24
+* Runtime: containerd or Docker
+* Namespace
+* [Storage class](https://kubernetes.io/docs/concepts/storage/storage-classes/)
+
+### Minimum Node Group Resource
+* 16 CPUs
+* 32 GB RAM
+* Storage with at least 3000 IOPS (>100GB recommended)
+
+### Permissions
+When Arthur platform is installed, Kubernetes RBAC resources are created to allow the Admin Console to manage the application.
+The kubectl-ing user who installs Arthur must have the wildcard privileges in the cluster.
+
+Refer to [this documentation](https://kots.io/vendor/packaging/rbac/#reference-objects) for the ClusterRole and ClusterRoleBinding that 
+will be created for the Admin Console.
+
+### Components
+* Prometheus
+* Ingress Controller (Nginx or Ambassador)
+* Kubernetes Metrics Server
+* Velero with Restic (Optional for managed backup and restore feature)
+  
+For Airgapped installation only:
+* An existing private container registry
+* Existing private Python registries (PyPI, Anaconda) - only required for the model explanation feature
+
+## VM Install
+
+### Minimum Server Resource
+* 16 CPUs
+* 32 GB RAM
+* Storage with at least 3000 IOPS (>100GB recommended)
+
+### Supported Operating Systems
+The latest versions of the following Linux operating systems are supported.
+
+* Ubuntu
+* RHEL
+
+Please do the following before running the installer on your VM for a smoother deployment experience:
+* If SELinux is enabled, set it to the permissive mode
+* Make sure the VM doesn't have any container runtime pre-installed, such as Docker or containerd
+
+### Ports for High Availability Configuration
+* TCP ports 2379, 2380, 6443, 6783, 10250, 10251 and 10252 open between cluster nodes
+* UDP ports 6783 and 6784 open between cluster nodes
+
+## Firewall Configurations
+### Ingress
+The TCP port 443 is the only entry point that Arthur exposes.
+
+### Egress 
+The platform requires access to any integrations (e.g. SMTP, IdP) as well as externalized components (e.g. Postgres, S3).   
+
+#### For Airgap Installation
+Your private container and Python registries must be accessible.
+
+(requirements_for_online_installation)=
+#### For Online Installation
+Access to container images and deployment manifest files from the below public registries are required.
+
+| Host                   | Existing Cluster  | Embedded Cluster |
+| ---------------------- | ----------------- | -----------------|
+| Docker Hub             | Required          | Required         |
+| proxy.replicated.com   | Required          | Required         |	
+| replicated.app         | Required          | Required         |
+| k8s.kurl.sh            | Not Required      | Required         |
+| amazonaws.com          | Not Required      | Required         |

files/arthur-docs-markdown/platform-management/installation/vm-installation/index.md.txt

@@ -0,0 +1,21 @@
+# Virtual Machine Installation
+
+This section covers the steps required for installing Arthur on a virtual machine. We have included seperate steps required for online and airgapped installations.
+
+## {doc}`vm_install_online`
+This page shows the basic process of installing Arthur on a virtual machine.
+
+## {doc}`vm_install_airgapped`
+This page shows the basic process of installing Arthur on an airgapped virtual machine.
+
+## {doc}`vm_install_airgapped_cli`
+This page shows the basic process of installing Arthur on an airgapped virtual machine via a command-line interface.
+
+```{toctree}
+:maxdepth: 1
+:hidden:
+
+Online <vm_install_online>
+Airgapped <vm_install_airgapped>
+Airgapped CLI <vm_install_airgapped_cli>
+```

files/arthur-docs-markdown/platform-management/installation/vm-installation/vm_install_airgapped.md.txt

@@ -0,0 +1,49 @@
+# Airgap Virtual Machine (VM) Install
+
+Go to the download portal using the URL and the password provided by Arthur.
+
+Select the "Embedded cluster" option. 
+![.png image](/_static/images/platform/download_portal_airgap_embedded.png)
+
+Click the “Download license” button to download your license in YAML file.
+
+Download the "Latest kURL embedded install" and the "Latest Arthur Airgap bundle".
+
+## Preparing the embedded cluster
+Arthur leverages Kubernetes as the base. This step installs the base Kubernetes cluster and Arthur's Admin Console 
+on your VM with a single CLI command.
+
+First, upload the kURL embedded install bundle on your VM instance.
+Example:
+```shell
+scp -i mykey.pem ~/Downloads/arthur.tar.gz ubuntu@hostname:arthur.tar.gz
+```
+
+Unpack the bundle and install the embedded Kubernetes cluster on your VM instance.
+```shell
+tar xvf arthur.tar.gz
+cat install.sh | sudo bash -s airgap
+```
+
+Save the output from the install which includes the Kotsadm Admin Console URL and the password.
+You now have a K8s cluster, kubectl CLI, and the Admin Console installed on your VM.
+
+## Deploying the application to the embedded cluster
+Load the Admin Console UI on port 8800 from your browser using the Kotsadm URL and the password you recorded earlier.
+Follow the instructions on the Admin Console to complete your installation by providing the `arthur-x.x.x.airgap` bundle and necessary configurations.
+
+Monitor the Admin Console dashboard for the application status to become **Ready**.
+
+![.png image](/_static/images/platform/dashboard.png)
+
+To see the progress of the deployment, monitor the deployment status with `kubectl` CLI on the VM:
+```shell
+# Reload your shell if you haven't
+bash -l
+
+kubectl get deployment,statefulset,pod
+```
+
+If anything is showing Pending, it is likely you need to add more/bigger nodes to your cluster.
+
+**Note:** You may also follow the instructions {doc}`here <vm_install_airgapped_cli>` to install the Admin Console and Arthur app programmatically using the CLI only.

files/arthur-docs-markdown/platform-management/installation/vm-installation/vm_install_airgapped_cli.md.txt

@@ -0,0 +1,24 @@
+# Airgap Virtual Machine (VM) Install with CLI
+
+If you prefer to install programmatically using CLI only, follow the steps below.
+
+Upload the license file and the `arthur-x.x.x.airgap` bundle on your VM instance.
+
+Example:
+```shell
+scp -i mykey.pem ~/Downloads/Test\ Customer.yaml ubuntu@hostname:license.yaml
+scp -i mykey.pem ~/Downloads/arthur-x.x.x.airgap ubuntu@hostname:arthur-x.x.x.airgap
+```
+
+Create a `config.yaml` file on the VM instance using {doc}`the configuration template </platform-management/reference/config_template>`.
+
+Run this install command from your VM's SSH session:
+
+```shell
+    kubectl kots install arthur \
+      --airgap-bundle ./arthur-x.x.x.airgap \
+      --license-file ./license.yaml \
+      --config-values ./config.yaml \
+      --namespace arthur \
+      --shared-password [The Kotsadm password you saved earlier]
+```

files/arthur-docs-markdown/platform-management/installation/vm-installation/vm_install_online.md.txt

@@ -0,0 +1,41 @@
+# Online Virtual Machine (VM) Install
+
+Go to the download portal using the URL and the password provided by Arthur.
+
+Click the "Download license" button to download your license in YAML file.
+
+SSH into your virtual machine (VM) and run the command below to install the Admin Console:
+```shell
+curl -sSL https://k8s.kurl.sh/arthur | sudo bash
+```
+
+Login to the Admin Console at `<yourhost>:8800` using the provided password in the install output. 
+
+Follow the instruction to set up your secure connection with TLS certificate.
+
+Upload your license file.
+![.png image](/_static/images/platform/license.png)
+
+Provide your configurations.
+![.png image](/_static/images/platform/configure.png)
+
+Review the preflight checks to make sure that your machine meets the minimum requirements before you proceed with the installation.
+
+Monitor the dashboard for the application status to become **Ready**.
+
+![.png image](/_static/images/platform/dashboard.png)
+
+To see the progress of the deployment, monitor the deployment status with `kubectl` CLI:
+```shell
+# Reload your shell if you haven't
+bash -l
+
+kubectl get deployment,statefulset,pod -n <yournamespace>
+```
+
+If anything is showing Pending, it is likely you need to add more/bigger nodes to your cluster.
+
+When using `kubectl`, you might run into a permission issue loading the `kubernetes/admin.conf` file. Please remediate it by running the command below.
+```shell
+sudo chmod +r /etc/kubernetes/admin.conf
+```

files/arthur-docs-markdown/platform-management/ongoing-maintenance/audit_log.md.txt

@@ -0,0 +1,112 @@
+# Audit Log
+
+The Arthur platform has the ability to produce an audit log of all calls to sensitive endpoints that include
+models, organizations, RBAC, and uploading / modifying data. 
+
+## Event Format 
+
+Each event in the audit log has the following fields:
+
+| Field            | Type           | Notes                                                                                                                                                                          |
+|------------------|----------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| event_category   | string         | A description of the overarching category for this event. See the table below for a breakdown of the various categories.                                                       |
+| event_type       | string         | An explanation of what kind of event occurred within the event_category. See the table below for a breakdown of the various types.                                             |
+| event_id         | string         | A unique ID for this event, currently in UUID format but this may change in the future.                                                                                        |
+| timestamp        | [string, int]  | A timestamp in either Unix Epoch millisecond integer format or [RFC 3339](https://datatracker.ietf.org/doc/html/rfc3339) string format, depending on the point of integration. |
+| organization_id  | [string, null] | A string UUID of the organization if there is one associated with the event.                                                                                                   |
+| model_id         | [string, null] | A string UUID of the model if there is one associated with the event.                                                                                                          |
+| user_id          | [string, null] | A string ID of the user if there is one associated with the event.                                                                                                             |
+| user_type        | [string, null] | A string description of the kind of user if there is one associated with the event. This can be one of: `service-account`, `arthur-managed`, or `idp-managed`.                 |
+| http_path        | [string, null] | A string HTTP path of the request that triggered the event if one exists.                                                                                                      |
+| http_method      | [string, null] | A string HTTP method of the request that triggered the event if one exists.                                                                                                    |
+| http_status_code | [int, null]    | An integer HTTP status code of the request that triggered the event if one exists.                                                                                             |
+
+## Logged Endpoints
+
+When enabled, Audit Logging will track all requests made to the following endpoints and set the Event Category and Event Type respectively in the audit log events.
+
+| Endpoint                              | Method | Event Category                | Event Type                      |
+|---------------------------------------|--------|-------------------------------|---------------------------------|
+| /organizations                        | POST   | events.arthur.ai/organization | created                         |
+| /organizations/{organization_id}      | DELETE | events.arthur.ai/organization | deleted                         |
+| /models                               | POST   | events.arthur.ai/model        | created                         |
+| /models/{model_id}                    | PUT    | events.arthur.ai/model        | updated                         |
+| /models/{model_id}                    | DELETE | events.arthur.ai/model        | deleted                         |
+| /alerts/{alert_id}/notifications      | POST   | events.arthur.ai/alert        | created                         |
+| /models/{model_id}/inferences         | POST   | events.arthur.ai/ingestion    | inference_data_received         |
+| /models/{model_id}/inferences         | PATCH  | events.arthur.ai/ingestion    | ground_truth_data_received      |
+| /models/{model_id}/inferences/file    | POST   | events.arthur.ai/ingestion    | inference_data_received         |
+| /models/{model_id}/reference_data     | POST   | events.arthur.ai/ingestion    | reference_data_received         |
+| /models/{model_id}/batches/{batch_id} | PATCH  | events.arthur.ai/ingestion    | inference_data_batch_completed  |
+| /models/{model_id}/reference_data     | PATCH  | events.arthur.ai/ingestion    | reference_data_upload_completed |
+| /authorization/custom_roles           | POST   | events.arthur.ai/rbac         | updated                         |
+| /authorization/custom_roles           | DELETE | events.arthur.ai/rbac         | updated                         |
+
+A more thorough description of these endpoints is available at our [API documentation](https://docs.arthur.ai/api-documentation/v3-api-docs.html).
+
+## Integration with EventBridge
+
+The on-prem installation provides support for shipping the Audit Log to AWS EventBridge. In order to configure this, you will need the following:
+
+- **Bus Name**: Required. The name of the EventBridge bus. This should not be the full ARN of the bus.
+- **Region**: Required. This is the AWS region where your EventBridge bus is located. 
+- **Source**: Optional. This value will be added to the EventBridge events "source" for all events. This defaults to "arthur-audit-log".
+- **Detail Type**: Optional. This value will be added to the EventBridge events "detail-type" for all events. This defaults to "events.arthur.ai".
+
+An example of the events that are written to EventBridge look like the following
+(this was captured via an EventBridge to CloudWatch Log Group rule and target):
+```json
+{
+    "version": "0",
+    "id": "b87f2a3a-6be1-e1d9-bc94-720d60e0a9d8",
+    "detail-type": "events.arthur.ai",
+    "source": "arthur-audit-log",
+    "account": "1234567890",
+    "time": "2022-07-21T22:07:00Z",
+    "region": "us-east-2",
+    "resources": [],
+    "detail": {
+        "event_type": "created",
+        "event_category": "events.arthur.ai/model",
+        "event_id": "da2ec82d-f581-4e72-bb66-fc82504f2a7e",
+        "timestamp": "2022-07-21T22:06:59.683+0000",
+        "organization_id": "d579359a-7259-4397-a08b-3e36c212350f",
+        "model_id": "a950c9ad-6a1e-4042-8e47-461d13072da5",
+        "user_id": "df3fe374-26d7-4bd8-bf62-e04a6e078e2b",
+        "user_type": "arthur-managed",
+        "http_path": "/api/v3/models",
+        "http_method": "POST",
+        "http_status_code": 200
+    }
+}
+```
+
+### Configuration
+
+The EventBridge integration can be enabled on the Admin Console Config Page by:
+1. Checking "Show Other Advanced Options" under the Other Advanced Options section
+2. After that is checked, a new section will appear called "Audit Logging"
+3. Check "Enable Audit Log"
+4. Next a choice of persistence methods appears. Choose "AWS EventBridge"
+5. Fill out the "Bus Name", "Region", "Event Source", and "Detail Type" fields that appear.
+6. Click "Save config" and deploy the updated version
+
+### Required IAM Permissions
+
+In order to send events to AWS EventBridge, the Arthur IAM credentials or role will require the `events:PutEvents` 
+permission. Here is an example policy that grants that permission on a EventBridge bus called `arthur-events` in the 
+`us-east-2` region, in the `0123456789` AWS account.
+
+```json
+{
+    "Statement": [
+        {
+            "Action": "events:PutEvents",
+            "Effect": "Allow",
+            "Resource": "arn:aws:events:us-east-2:0123456789:event-bus/arthur-events",
+            "Sid": ""
+        }
+    ],
+    "Version": "2012-10-17"
+}
+```

files/arthur-docs-markdown/platform-management/ongoing-maintenance/backup.md.txt

@@ -0,0 +1,845 @@
+# Arthur Platform Backup and Restore
+
+## Contents
+
+* Warnings
+* Overview
+    * Overview - clickhouse-backup
+    * Overview - Velero
+    * Overview - Arthur (Argo)Workflows
+    * Overview - S3
+* Pre-requisites
+* Installing - Velero
+    * Setup ServiceAccount with IAM roles for Backup S3 Bucket
+    * Setup CRDs + Cluster-level permissions + Backup Infrastructure
+    * Confirm Velero is installed and configured correctly
+    * Configure the Backup Storage Destination to Point to S3
+* Configuring - clickhouse-backup remote storage
+* Backing up
+* Restoring
+* Appendix
+    * Running the Velero CLI
+    * Working with Velero - Backup
+    * Working with Velero - Restore
+    * Backup Architecture
+
+## WARNINGS
+
+### PLEASE READ - FOLLOW THESE INSTRUCTIONS EXACTLY
+
+These instructions have been tested as written. If you find they do not work for your use-case, please reach out to Arthur Support before modifying them. We cannot guarantee reliable operation if these instructions are not followed exactly as written.
+
+### PLEASE READ - TAKE CARE WHEN RESTORING INTO A NEW CLUSTER
+
+When restoring into a new cluster, you must ensure that the new cluster is unable to communicate with any services or data store in the old cluster.
+
+If you took a backup on cluster "Apple", and performed a restore into cluster "Banana", cluster "Banana" must point to its own RDS Instance, ClickHouse Database, and Kafka Store (note: it is ok if clusters share an S3 bucket, but not ideal).
+
+To ensure this, you must re-configure via the Admin Interface when restoring into a new cluster. Failure to do this **WILL CAUSE DATA CORRUPTION** on both clusters that is unrecoverable.
+
+### PLEASE READ - ENSURE CONSISTENCY WITH BACKUPS
+
+If you are either manually taking a backup, or scheduling a backup, you **MUST** take a backup of of the full platform. You **CANNOT** use a ClickHouse snapshot taken at midnight with a RDS snapshot taken at 0400 AM (or any other time). All backup operations must be performed at the same time, and when restoring, the data you are using must all belong to the same backup operation. This is to ensure data consistency across the different data stores. **IGNORING THIS WILL CAUSE DATA CORRUPTION**.
+
+## Overview
+
+The overall backup and restore process for the Arthur Platform is as follows:
+
+* Backup a cluster
+	* Take a backup of ClickHouse Data
+	* Take a backup of Kubernetes Deployment State and Persistent Volumes
+		* Enrichments infrastructure
+			* Model Servers
+			* Data Pipeline Services
+			* Enrichment / Delete Enrichment Workflows
+		* Kafka Deployment State and EBS Volumes (using EBS Snapshots)
+	* Take a backup of RDS Postgres
+* Restore the cluster
+	* Restore RDS Postgres
+	* Update configuration and install the platform
+	* Restore ClickHouse Data
+	* Restore the Kafka Deployment State and Persistent Volumes
+	* Restore Enrichments infrastructure
+	* Restore Workflows
+* Smoke Tests and Validation
+
+### Overview - clickhouse-backup
+
+The Arthur Platform stores inference data, data built from the enrichments pipeline, reference and ground truth data in ClickHouse. ClickHouse is an open-source OLAP Database which enables SQL-like query execution, replication, sharding and many additional features.
+
+To backup ClickHouse, the Arthur Platform uses a tool called [clickhouse-backup](https://github.com/AlexAkulov/clickhouse-backup). clickhouse-backup is a sidecar-container included on the ClickHouse pods and is responsible for taking backups, performing restores, and coordinating with remote storage (in this case S3) to store and retrieve backups. clickhouse-backup uses built-in functionality of ClickHouse to take backups and perform restores.
+
+
+### Overview - Velero
+
+The Arthur Platform uses [Velero](https://velero.io/) as its Backup and Restore tool. Velero is an industry-standard, battle-tested tool for backing up Kubernetes Resources as well as Persistent Volumes.
+
+Arthur uses velero to backup most namespaced kubernetes resources, as well as the EBS Volume Snapshot backups for each PersistentVolumes claimed by the StatefulSets (eg: via PVCs).
+
+Backup data (not including EBS Volume Snapshots) is stored in an S3 bucket which is accessible via a ServiceAccount that is provisioned for the Backup and Restore agent. Backup and Restores are managed by Velero using Kubernetes Custom Resource Definitions (CRDs), which are consumed by the Velero Backup Controller and Restic Agents.
+
+Velero has a feature which also allows backups to be scheduled, using a cron-like configuration. It also provides `ServiceMonitors` which expose metrics via Prometheus, so that operators can monitor backup and restore status and set up alerts for when backups or restores fail.
+
+### Overview - Arthur (Argo)Workflows
+
+The Arthur Platform uses [Argo Workflows](https://argoproj.github.io/argo-workflows/) as a workflow orchestration engine for running certain jobs. Argo installs a handful of Custom Resource Definitions (CRDs) which enable the Argo Workflow services to schedule, execute and update these jobs.
+
+Workflows are dynamically managed, meaning that their definitions are not stored in the Arthur installer script. The Backup and Restore operation accounts for this by treating restoration of Workflows on a case-by-case basis, as follows:
+
+* Enrichments and Delete Enrichments workflows
+	* These workflows are created to create and tear-down infrastructure necessary for processing enrichments data (eg: kafka topics, pods which manage the data pipeline for enrichments, etc.)
+	* These workflows are idempotent and safe to recover
+	* Therefore, these workflows are backed up and restored just like any other Kubernetes Resource during the backup stage
+* Batch workflows
+	* These workflows are created to manage batch jobs, which are used by clients when uploading large data files to models (inferences and/or ground truths).
+	* These workflows are sometimes safe to recover
+	* Therefore, these workflows are restored selectively based on what state they were in when the backup was taken
+		* Workflows for which Arthur received all of the data from the client are resumed by manually re-submitting them (this is done via an Administrative HTTP endpoint that needs to manually be called)
+		* Workflows for which Arthur did not receive all the data from the client will need to be re-submitted. Operators restoring the cluster will need to reach out to affected clients to communicate that their batch workflows should be re-submitted.
+* Reference and Cron Workflows
+	* Reference Workflows are created for monitoring the upload of reference datasets to S3
+		* Reference datasets that were in-flight during a backup will need to be re-uploaded with the SDK.
+	* Cron Workflows are scheduled workflows which perform some regular processing (eg: triggering alerts for non-batch inferences)
+		* Cron Workflows are meant to be run on a regular schedule. It is safe to wait for the next workflow to be triggered, and therefore, these workflows are not backed up nor restored.
+
+### Overview - S3
+
+The Arthur Platform uses AWS S3 as object storage for storing inference data, reference data, as well as data and trained models for the enrichments pipeline.
+
+Arthur recommends ensuring that the AWS S3 bucket used for this storage is configured with Live Cross-Region Replication so that objects are available in the event of an AWS region outage.
+
+The Arthur Backup solution does not manage consistency with the S3 bucket and other backup data.
+The data in S3 is only used in conjuction with data that is stored in Postgres (eg: model definitions), so it's ok if there's data in S3 that isn't represented in Postgres.
+Therefore, the S3 bucket for a cluster will always reflect the most up-to-date state, regardless of when a backup was taken.
+
+To read more about S3 Bucket Replication, check out the AWS Documentation:
+* https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html
+
+## Pre-requisites
+
+
+The following items must be configured specifically in this capacity in order to use Arthur's Backup and Restore capabilities:
+
+1. Arthur must be configured using external object storage, specifically, S3
+2. The access to external storage must be configured using IRSA Annotations
+3. In order to use IRSA Annotations, the cluster must be deployed using Amazon EKS
+
+If the following are not true/possible for your deployment, please reach out to Arthur Support so we can discuss.
+
+
+
+## Installing - Velero
+
+The only component that needs to be installed separately from Arthur to perform backup and restores is Velero. Below, instructions are provided for setting up Velero to store backups in S3 using Secret and Access keys.
+
+The general overview of the installation is as follows:
+
+1. Create the Velero configuration
+	1. Create the policy for accessing the S3 Bucket and taking EBS Snapshots and attach to an IAM User
+	2. Generate the Secret and Access keys for the IAM User
+	3. Create a Velero-specific credentials file
+4. Install Velero
+5. Confirm Velero is installed and configured correctly
+6. Configure the Backup Storage Destination to Point to S3
+
+### Create the Velero Configuration
+
+The instructions here are taken from the Velero AWS Plugin Documentation, which can be found in Option 1 here:
+https://github.com/vmware-tanzu/velero-plugin-for-aws#setup
+
+1. Create the IAM user:
+
+    ```bash
+    aws iam create-user --user-name velero
+    ```
+
+    If you'll be using Velero to backup multiple clusters with multiple S3 buckets, it may be desirable to create a unique username per cluster rather than the default `velero`.
+
+2. Attach policies to give `velero` the necessary permissions:
+
+    ```
+    cat > velero-policy.json <<EOF
+    {
+        "Version": "2012-10-17",
+        "Statement": [
+            {
+                "Effect": "Allow",
+                "Action": [
+                    "ec2:DescribeVolumes",
+                    "ec2:DescribeSnapshots",
+                    "ec2:CreateTags",
+                    "ec2:CreateVolume",
+                    "ec2:CreateSnapshot",
+                    "ec2:DeleteSnapshot"
+                ],
+                "Resource": "*"
+            },
+            {
+                "Effect": "Allow",
+                "Action": [
+                    "s3:GetObject",
+                    "s3:DeleteObject",
+                    "s3:PutObject",
+                    "s3:AbortMultipartUpload",
+                    "s3:ListMultipartUploadParts"
+                ],
+                "Resource": [
+                    "arn:aws:s3:::${BUCKET}/*"
+                ]
+            },
+            {
+                "Effect": "Allow",
+                "Action": [
+                    "s3:ListBucket"
+                ],
+                "Resource": [
+                    "arn:aws:s3:::${BUCKET}"
+                ]
+            }
+        ]
+    }
+    EOF
+    ```
+    ```bash
+    aws iam put-user-policy \
+      --user-name velero \
+      --policy-name velero \
+      --policy-document file://velero-policy.json
+    ```
+
+3. Create an access key for the user:
+
+    ```bash
+    aws iam create-access-key --user-name velero
+    ```
+
+    The result should look like:
+
+    ```
+    {
+      "AccessKey": {
+            "UserName": "velero",
+            "Status": "Active",
+            "CreateDate": "2017-07-31T22:24:41.576Z",
+            "SecretAccessKey": <AWS_SECRET_ACCESS_KEY>,
+            "AccessKeyId": <AWS_ACCESS_KEY_ID>
+      }
+    }
+    ```
+
+4. Create a Velero-specific credentials file (`credentials-velero`) in your local directory:
+
+    ```bash
+    [default]
+    aws_access_key_id=<AWS_ACCESS_KEY_ID>
+    aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
+    ```
+
+    where the access key id and secret are the values returned from the `create-access-key` request.
+
+### Install Velero
+
+To install Velero, first install the Velero CLI. Instructions for how to do this can be found on the Velero Documentation site: https://velero.io/docs/v1.10/basic-install/#install-the-cli
+
+Once the Velero CLI is installed, you can use this to install Velero on the cluster. Please ensure that your kubeconfig is pointing to the cluster, as this is what it used by the Velero CLI to communicate with the cluster.
+
+To install Velero, use the following command, ensuring that the path to the credentials file you generated in the last step is provided correctly:
+
+```
+velero install \
+    --provider aws \
+    --plugins velero/velero-plugin-for-aws:v1.6.0 \
+    --bucket $BUCKET \
+    --backup-location-config region=$REGION \
+    --snapshot-location-config region=$REGION \
+    --secret-file ./credentials-velero
+```
+
+### Confirm Velero is installed and configured correctly
+
+To confirm that Velero is installed and configured correctly:
+
+1. Open the Kots Admin Interface and navigate to the "Snapshots" tab
+2. Click the "Check for Velero" button (see the screenshot below)
+
+![Check for Velero](/_static/images/velero-configuration-check.png)
+
+### Configure the Backup Storage Destination to Point to S3
+
+The final step in configuring Velero is to configure the Backup Storage Destination to point to the S3 Bucket where the backups will be stored.
+
+To do this, add a new "Backup Storage Destination" in the Admin Interface and fill in the details for the S3 Bucket and for the Secret and Access keys.
+
+## Configuring - clickhouse-backup remote storage
+
+Before continuing, please ensure that your cluster is setup and configured as described in the "Pre-requisites" section above.
+
+Configuring clickhouse-backup to store backups in remote storage (eg: S3) can be done in the Kots Admin Interface.
+
+If you've performed the configuration steps as mentioned in the "Pre-requisites" section, you should see the "Enable Olap Database Backup Capabilites" option in the "Olap Database" section (see screenshot)
+
+![Configure Olap Backups](/_static/images/clickhouse-backup.png)
+
+Ensure that:
+
+1. The configuration that points to the bucket is correct
+    * The Bucket Name
+    * The Bucket Region
+2. The ServiceAccount is the same ServiceAccount that you've configured with the IRSA Annotation (if you are not sure, enter the default value)
+3. The IAM Role that you are using for the IRSA Annotation has the appropriate permissions to read/write/list from the S3 bucket
+    * Note - you can use the same permissions that are described in the Velero Section, just be sure to update the bucket ARN that the permissions apply to
+4. The S3 Path is where you want to be storing backups
+
+Once you've configured clickhouse-backup, you can validate that the configuration is correct by trying to take a backup. Refer to the "Take a backup of ClickHouse Data" section below on how to do this.
+
+## Backing up
+
+Note that you can use the following script to run all of these steps together. Read the `Appendix: Running the Velero CLI` for instructions on how to run velero.
+
+```bash
+#!/bin/bash
+
+set -euo pipefail
+IFS=$'\n\t'
+
+# You need to configure this by getting the name of the storage location
+# using the velero CLI
+# eg: `velero backup-location get`
+storage_location="Put your storage location here"
+
+backup_date=$(DATE +%Y-%m-%d-%H-%M-%S);
+name=arthur-backup-$backup_date
+echo "Creating a new backup with name $name"
+
+echo "Taking a backup of CH data"
+kubectl create job $name-clickhouse-backup \
+    --from=cronjob/clickhouse-backup-cronjob
+ch_backup_jobname=$(kubectl get jobs -o name | grep "$name-clickhouse-backup")
+kubectl wait $ch_backup_jobname \
+    --for=condition=complete \
+    --timeout=30m
+
+echo "Taking a backup of the enrichments infrastructure"
+velero backup create $name-enrichments \
+    --namespace=arthurai \
+    --include-namespaces=arthurai \
+    --selector='component in (kafka-mover-init-connector, model_server)' \
+    --include-resources=deployments,services \
+    --exclude-resources=clusterrolebindings.rbac.authorization.k8s.io,clusterroles.rbac.authorization.k8s.io,controllerrevisions.apps,endpointslices.discovery.k8s.io,customresourcedefinitions.apiextensions.k8s.io,secrets,configmaps \
+    --storage-location=$storage_location \
+    --wait
+
+echo "Taking a backup of workflows"
+velero backup create $name-workflows \
+    --namespace=arthurai \
+    --include-namespaces=arthurai \
+    --include-resources=workflows \
+    --exclude-resources=clusterrolebindings.rbac.authorization.k8s.io,clusterroles.rbac.authorization.k8s.io,controllerrevisions.apps,endpointslices.discovery.k8s.io,customresourcedefinitions.apiextensions.k8s.io,secrets,configmaps \
+    --storage-location=$storage_location \
+    --wait
+
+echo "Taking a backup of Kafka/Kafka-ZK StatefulSets, their EBS Volumes, and related components"
+velero backup create $name-stateful-sets \
+    --namespace=arthurai \
+    --include-namespaces=arthurai \
+    --selector='app in (cp-zookeeper,cp-kafka)' \
+    --exclude-resources=clusterrolebindings.rbac.authorization.k8s.io,clusterroles.rbac.authorization.k8s.io,controllerrevisions.apps,endpointslices.discovery.k8s.io,customresourcedefinitions.apiextensions.k8s.io \
+    --storage-location=$storage_location \
+    --wait
+
+echo "Taking a backup of the RDS database"
+aws rds create-db-cluster-snapshot \
+    --db-cluster-snapshot-identifier $name-snapshot \
+    --db-cluster-identifier RDS_DB_NAME \
+    --profile AWS_PROFILE_NAME \
+    --region AWS_REGION
+```
+
+### Take a backup of ClickHouse Data
+
+By default, the Arthur Platform ships with a Kubernetes CronJob which takes a ClickHouse Backup each day at midnight.
+
+Please see the warning `PLEASE READ - ENSURE DATA CONSISTENCY` at the top if attempting to take a manual backup.
+
+To take a manual backup of ClickHouse Data, you can run the following commands:
+
+```bash
+$ kubectl get cronjobs -n arthurai
+NAME                              SCHEDULE    SUSPEND   ACTIVE   LAST SCHEDULE   AGE
+arthurai-cron-workflow-cron-job   1 0 * * *   False     0        14h             2d18h
+arthurai-model-health-cron-job    5 * * * *   False     0        20m             2d18h
+clickhouse-backup-cronjob         0 0 * * *   False     0        14h             2d18h
+$ kubectl create job clickhouse-backup --from=cronjob/clickhouse-backup-cronjob -n arthurai
+job.batch/clickhouse-backup created
+$ kubectl get jobs -n arthurai
+NAME                                       COMPLETIONS   DURATION   AGE
+clickhouse-backup-cronjob-27735840         1/1           8m35s      14m
+```
+
+### Take a backup of Enrichments infrastructure and Enrichment / Delete Enrichment Workflows
+
+The Arthur Platform uses Velero to take a backup of the Enrichments Infrastructure, as well as the Enrichments workflows. The backup is orchestrated manually and requires running a command.
+
+The Enrichments infrastructure and Enrichment Workflows are orchestrated as separate backups and will require running 2 separate commands.
+
+To take a manual backup of the Enrichments infrastructure, run the following commands:
+
+```bash
+$ backup_date=$(DATE +%Y-%m-%d-%H-%M-%S);
+$ name=arthur-backup-$backup_date
+$ velero backup create $name-enrichments \
+    --namespace=arthurai \
+    --include-namespaces=arthurai \
+    --selector='component in (kafka-mover-init-connector, model_server)' \
+    --include-resources=deployments,services \
+    --exclude-resources=clusterrolebindings.rbac.authorization.k8s.io,clusterroles.rbac.authorization.k8s.io,controllerrevisions.apps,endpointslices.discovery.k8s.io,customresourcedefinitions.apiextensions.k8s.io,secrets,configmaps \
+    --storage-location=$storage_location \
+    --wait
+```
+
+To take a manual backup of the Enrichments Workflows, run the following commands:
+
+```bash
+$ backup_date=$(DATE +%Y-%m-%d-%H-%M-%S);
+$ name=arthur-backup-$backup_date
+velero backup create $name-workflows \
+    --namespace=arthurai \
+    --include-namespaces=arthurai \
+    --include-resources=workflows \
+    --exclude-resources=clusterrolebindings.rbac.authorization.k8s.io,clusterroles.rbac.authorization.k8s.io,controllerrevisions.apps,endpointslices.discovery.k8s.io,customresourcedefinitions.apiextensions.k8s.io,secrets,configmaps \
+    --storage-location=$storage_location \
+    --wait
+```
+
+### Take a backup of Kafka Deployment State and EBS Volumes (using EBS Snapshots)
+
+The Arthur Platform uses Velero to take a backup of the Kafka Deployment State and EBS Volumes. The backup is orchestrated manually and requires running a command.
+
+To take a manual backup of Kafka, run the following commands:
+
+```bash
+$ backup_date=$(DATE +%Y-%m-%d-%H-%M-%S);
+$ name=arthur-backup-$backup_date
+$ velero backup create $name-zookeeper \
+    --namespace=arthurai \
+    --include-namespaces=arthurai \
+    --selector='app in (cp-zookeeper,cp-kafka)' \
+    --exclude-resources=clusterrolebindings.rbac.authorization.k8s.io,clusterroles.rbac.authorization.k8s.io,controllerrevisions.apps,endpointslices.discovery.k8s.io,customresourcedefinitions.apiextensions.k8s.io \
+    --storage-location=$storage_location \
+    --wait
+```
+
+If you are unable to run the Velero CLI locally, you can execute it from the Velero container. See the section **Executing on the Velero Backup Controller Pod** for more details.
+
+
+### Take a backup of RDS Postgres
+
+The script above provides the command line instructions for taking a backup of a RDS Database (also copied here). Please ensure that the values for the `db-cluster-identifier`, `profile` and `region` are filled in correctly.
+
+Note that the command is only compatible for a multi-region RDS Database. If you are using a single-region RDS Database, the command to use is `aws rds create-db-snapshot`.
+
+```bash
+aws rds create-db-cluster-snapshot \
+    --db-cluster-snapshot-identifier $name-snapshot \
+    --db-cluster-identifier RDS_DB_NAME \
+    --profile AWS_PROFILE_NAME \
+    --region AWS_REGION
+```
+
+For more information, please refer to the AWS Documentation:
+* Multi-region RDS instance:
+    * https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateMultiAZDBClusterSnapshot.html
+    * https://awscli.amazonaws.com/v2/documentation/api/latest/reference/rds/create-db-cluster-snapshot.html
+* Single-region RDS instance:
+    * https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateSnapshot.html
+    * https://awscli.amazonaws.com/v2/documentation/api/latest/reference/rds/create-db-snapshot.html
+
+## Restoring
+
+### Restore the RDS Postgres
+
+To restore RDS Postgres, use the following resource:
+https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_RestoreFromMultiAZDBClusterSnapshot.Restoring.html
+
+Please ensure that you correctly match the following configuration from the DB from which the snapshot was taken
+* The connection port
+* The VPC and Security Group IDs
+* DB Subnet Group
+* DB Instance Type
+* Any other configuration which might be overridden
+
+This operation might take a while and the DB must show as Available before installing the platform.
+
+### Update configuration and install the platform
+
+**NOTE - Before completing this action, please check the following:**
+
+1. The Restored RDS DB Instance is showing as "Available"
+2. All pods which were restored are showing as "Running"
+
+**NOTE - IMPORTANT -- Update configuration to point to the new DB Instance**
+
+It is **CRITICALLY IMPORTANT** to update your configuration to point to the newly restored DB Instance. Failure to complete this step **WILL CAUSE DATA CORRUPTION**.
+
+Please update the configuration in the "Meta Database" section of the Admin Interface to point to the newly restored DB instance.
+
+If your new cluster is routed via a different ingress URL, please also update this in the "Network" section of the configuration.
+
+Wait for the platform to come back online before proceeding to the next steps. All Deployments and StatefulSets should be completely stood up (eg: all Pods should be ready and "Running") and all Jobs should be "Completed".
+
+
+### Restore ClickHouse Data
+
+The Arthur Platform ships with a Kubernetes CronJob that executes a ClickHouse restore that is scheduled to never run (eg: it's scheduled to run on February 31st, which is an invalid date).
+
+To run a ClickHouse data restore, execute the following commands:
+
+First, we need to get the name of the clickhouse-backup that coincides with the kafka/enrichments/workflow backups that you are restoring:
+
+```bash
+$ kubectl get pods | grep olap
+chi-olap-installation-arthur-0-0-0                           2/2     Running     0          6d23h
+chi-olap-installation-arthur-0-1-0                           2/2     Running     0          6d23h
+chi-olap-installation-arthur-0-2-0                           2/2     Running     0          6d23h
+olap-installation-zookeeper-0                                3/3     Running     0          6d23h
+olap-installation-zookeeper-1                                3/3     Running     0          6d22h
+olap-installation-zookeeper-2                                3/3     Running     0          6d23h
+olap-operator-8c867ddff-kc22x                                2/2     Running     0          6d23h
+$ kubectl exec chi-olap-installation-arthur-0-0-0 -c backup -- clickhouse-backup list
+2022/10/04 16:03:20.255378  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER'
+2022/10/04 16:03:20.257582  info SELECT * FROM system.disks;
+2022/10/04 16:03:20.267149  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros'
+2022/10/04 16:03:20.271359  info SELECT * FROM system.macros
+chi-olap-installation-arthur-0-0-arthur-clickhouse-backup-2022-09-26-19-50-43   3.64MiB   02/10/2022 00:00:16   remote      tar, regular
+chi-olap-installation-arthur-0-1-arthur-clickhouse-backup-2022-09-26-19-50-43   3.64MiB   02/10/2022 00:00:16   remote      tar, regular
+chi-olap-installation-arthur-0-2-arthur-clickhouse-backup-2022-09-26-19-50-43   3.64MiB   02/10/2022 00:00:16   remote      tar, regular
+```
+
+Note that from the above backup, the backup names are in the following format:
+`$CLICKHOUSE_NODE_NAME-$ARTHUR_BACKUP_NAME`
+
+So for example, `chi-olap-installation-arthur-0-0-arthur-clickhouse-backup-2022-09-26-19-50-43` can be parsed into:
+* clickhouse node name: `chi-olap-installation-arthur-0-0`
+* arthur backup name: `arthur-clickhouse-backup-2022-09-26-19-50-43`
+
+Now we need to create the restoration job, and configure it to use the Arthur Backup Name from above:
+
+```bash
+$ kubectl create job --from=cronjob/clickhouse-restore-cronjob -o yaml clickhouse-restore --dry-run=client --save-config > clickhouse-restore.yaml
+
+# update the value of the `BACKUP_NAME` environment variable in the `clickhouse-restore.yaml` file
+# eg:
+#        - name: BACKUP_NAME
+#          value: "arthur-clickhouse-backup-2022-09-26-19-50-43"
+
+$ kubectl apply -f clickhouse-restore.yaml
+job.batch/clickhouse-restore created
+```
+
+### Restore the Kafka Deployment State and Persistent Volumes
+
+The Arthur Platform restores Kafka Deployment State and PVs using Velero.
+
+To execute a restore, run the following commands:
+
+```bash
+$ velero backup get -n arthurai | grep kafka
+NAME                                                    STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
+arthur-backup-2022-09-23t11.23.25-04.00-kafka           Completed   0        0          2022-09-23 11:24:37 -0400 EDT   27d       default            app in (cp-kafka,cp-zookeeper)
+$ velero restore create \
+    --from-backup "arthur-backup-2022-09-23t11.23.25-04.00-kafka" \
+    --namespace arthurai \
+    --restore-volumes=true \
+    --existing-resource-policy=update
+```
+
+Velero will update the Pod Specs and point to the PVs using the EBS Volume Snapshots and restore the kubernetes resources associated with Kafka.
+
+### Restore Enrichments infrastructure
+
+The Arthur Platform uses Velero to restore the Enrichments infrastructure.
+
+To restore, run the following commands:
+
+```bash
+$ velero backup get -n arthurai | grep enrichments
+NAME                                                    STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
+arthur-backup-2022-09-23t11.23.25-04.00-enrichments     Completed   0        0          2022-09-23 11:24:33 -0400 EDT   27d       default            component in (kafka-mover-init-connector,model_server)
+$ velero restore create \
+    --from-backup "arthur-backup-2022-09-23t11.23.25-04.00-enrichments" \
+    --namespace arthurai
+```
+
+### Restore Workflows
+
+Restoring workflows is a 2-step process:
+
+1. Restore the workflows from the Velero backup
+2. Restore Batch Workflows which are recoverable using an Arthur Admin Endpoint
+
+To restore workflows using the Velero backup, run the following commands:
+
+```bash
+$ velero backup get -n arthurai | grep workflows
+NAME                                                    STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
+arthur-backup-2022-09-23t11.23.25-04.00-workflows       Completed   0        0          2022-09-23 11:24:35 -0400 EDT   27d       default            <none>
+$ velero restore create \
+    --from-backup "arthur-backup-2022-09-23t11.23.25-04.00-workflows" \
+    --namespace arthurai
+```
+
+To restore Batch Workflows, run the following comamnds.
+
+In one terminal window, port-forward to the dataset-service:
+
+```bash
+$ kubectl get pods | grep dataset-service
+arthurai-dataset-service-69dd979d8c-72t7j                         1/1     Running            0          2d17h
+$ kubectl port-forward -n arthurai arthurai-dataset-service-69dd979d8c-72t7j 7899
+```
+
+Then, in another terminal window, run the following commands:
+
+```bash
+$ curl -k -XPOST https://localhost:7899/api/v1/workflows/batch/recover
+{"message":"success"}
+```
+
+### Smoke Tests and Validation
+
+The restore should now be complete. All data should be restored and in a consistent state as it was from when the backup was taken. Any data sent during or after the backup will need to be re-sent. Perform any validation/smoke-tests to ensure that the platform is operating.
+
+## Appendix
+
+### Running the Velero CLI
+
+Velero provides a Command-Line Interface (CLI) for taking backups and performing restores. The command line interface can be installed locally, or it can be invoked by `kubectl exec` on the Velero Backup Controller pod.
+
+#### Local Installation
+
+Read through the Velero Documentation for how to install velero on your platform:
+https://velero.io/docs/v1.9/basic-install/#install-the-cli
+
+Velero uses your KUBECONFIG file to connect to the cluster.
+
+```
+$ velero --help
+Velero is a tool for managing disaster recovery, specifically for Kubernetes
+cluster resources. It provides a simple, configurable, and operationally robust
+way to back up your application state and associated data.
+
+If you're familiar with kubectl, Velero supports a similar model, allowing you to
+execute commands such as 'velero get backup' and 'velero create schedule'. The same
+operations can also be performed as 'velero backup get' and 'velero schedule create'.
+
+Usage:
+  velero [command]
+
+Available Commands:
+  backup            Work with backups
+  backup-location   Work with backup storage locations
+  bug               Report a Velero bug
+  client            Velero client related commands
+  completion        Generate completion script
+  create            Create velero resources
+  debug             Generate debug bundle
+  delete            Delete velero resources
+  describe          Describe velero resources
+  get               Get velero resources
+  help              Help about any command
+  install           Install Velero
+  plugin            Work with plugins
+  restic            Work with restic
+  restore           Work with restores
+  schedule          Work with schedules
+  snapshot-location Work with snapshot locations
+  uninstall         Uninstall Velero
+  version           Print the velero version and associated image
+
+Flags:
+      --add_dir_header                   If true, adds the file directory to the header
+      --alsologtostderr                  log to standard error as well as files
+      --colorized optionalBool           Show colored output in TTY. Overrides 'colorized' value from $HOME/.config/velero/config.json if present. Enabled by default
+      --features stringArray             Comma-separated list of features to enable for this Velero process. Combines with values from $HOME/.config/velero/config.json if present
+  -h, --help                             help for velero
+      --kubeconfig string                Path to the kubeconfig file to use to talk to the Kubernetes apiserver. If unset, try the environment variable KUBECONFIG, as well as in-cluster configuration
+      --kubecontext string               The context to use to talk to the Kubernetes apiserver. If unset defaults to whatever your current-context is (kubectl config current-context)
+      --log_backtrace_at traceLocation   when logging hits line file:N, emit a stack trace (default :0)
+      --log_dir string                   If non-empty, write log files in this directory
+      --log_file string                  If non-empty, use this log file
+      --log_file_max_size uint           Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
+      --logtostderr                      log to standard error instead of files (default true)
+  -n, --namespace string                 The namespace in which Velero should operate (default "velero")
+      --skip_headers                     If true, avoid header prefixes in the log messages
+      --skip_log_headers                 If true, avoid headers when opening log files
+      --stderrthreshold severity         logs at or above this threshold go to stderr (default 2)
+  -v, --v Level                          number for the log level verbosity
+      --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging
+
+Use "velero [command] --help" for more information about a command.
+```
+
+#### Executing on the Velero Backup Controller Pod
+
+```
+$ kubectl exec velero-699dc869d4-r24bh -c velero -- /velero
+Velero is a tool for managing disaster recovery, specifically for Kubernetes
+cluster resources. It provides a simple, configurable, and operationally robust
+way to back up your application state and associated data.
+
+If you're familiar with kubectl, Velero supports a similar model, allowing you to
+execute commands such as 'velero get backup' and 'velero create schedule'. The same
+operations can also be performed as 'velero backup get' and 'velero schedule create'.
+
+Usage:
+  velero [command]
+
+Available Commands:
+  backup            Work with backups
+  backup-location   Work with backup storage locations
+  bug               Report a Velero bug
+  client            Velero client related commands
+  completion        Generate completion script
+  create            Create velero resources
+  debug             Generate debug bundle
+  delete            Delete velero resources
+  describe          Describe velero resources
+  get               Get velero resources
+  help              Help about any command
+  install           Install Velero
+  plugin            Work with plugins
+  restic            Work with restic
+  restore           Work with restores
+  schedule          Work with schedules
+  snapshot-location Work with snapshot locations
+  uninstall         Uninstall Velero
+  version           Print the velero version and associated image
+
+Flags:
+      --add_dir_header                   If true, adds the file directory to the header
+      --alsologtostderr                  log to standard error as well as files
+      --colorized optionalBool           Show colored output in TTY. Overrides 'colorized' value from $HOME/.config/velero/config.json if present. Enabled by default
+      --features stringArray             Comma-separated list of features to enable for this Velero process. Combines with values from $HOME/.config/velero/config.json if present
+  -h, --help                             help for velero
+      --kubeconfig string                Path to the kubeconfig file to use to talk to the Kubernetes apiserver. If unset, try the environment variable KUBECONFIG, as well as in-cluster configuration
+      --kubecontext string               The context to use to talk to the Kubernetes apiserver. If unset defaults to whatever your current-context is (kubectl config current-context)
+      --log_backtrace_at traceLocation   when logging hits line file:N, emit a stack trace (default :0)
+      --log_dir string                   If non-empty, write log files in this directory
+      --log_file string                  If non-empty, use this log file
+      --log_file_max_size uint           Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
+      --logtostderr                      log to standard error instead of files (default true)
+  -n, --namespace string                 The namespace in which Velero should operate (default "arthurai")
+      --skip_headers                     If true, avoid header prefixes in the log messages
+      --skip_log_headers                 If true, avoid headers when opening log files
+      --stderrthreshold severity         logs at or above this threshold go to stderr (default 2)
+  -v, --v Level                          number for the log level verbosity
+      --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging
+
+Use "velero [command] --help" for more information about a command.
+```
+
+### Working with Velero - Backup
+
+To take a backup of Arthur, you would invoke the CLI as follows.
+
+```bash
+$ velero backup create arthur-backup-$(date -Iseconds | tr ":" "." | tr "T" "t") \
+    --namespace=arthurai \
+    --include-namespaces=arthurai
+```
+
+The command above will create a Velero Backup Resource named `arthur-backup-<timestamp>`, which you can check using the Velero CLI:
+
+```bash
+$ velero backup get -n arthurai
+NAME                                      STATUS       ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
+arthur-backup-2022-08-25t09.59.35-04.00   Completed    0        0          2022-08-25 09:59:35 -0400 EDT   28d       default            <none>
+arthur-backup-2022-08-26t10.37.52-04.00   InProgress   0        0          2022-08-26 10:37:52 -0400 EDT   29d       default            <none>
+```
+
+For debugging a backup, you can access the backup's logs using the Velero CLI:
+
+```bash
+$ velero backup logs arthur-backup-2022-08-25t09.59.35-04.00 -n arthurai | head
+time="2022-08-25T13:59:35Z" level=info msg="Setting up backup temp file" backup=arthurai/arthur-backup-2022-08-25t09.59.35-04.00 logSource="pkg/controller/backup_controller.go:587"
+time="2022-08-25T13:59:35Z" level=info msg="Setting up plugin manager" backup=arthurai/arthur-backup-2022-08-25t09.59.35-04.00 logSource="pkg/controller/backup_controller.go:594"
+time="2022-08-25T13:59:35Z" level=info msg="Getting backup item actions" backup=arthurai/arthur-backup-2022-08-25t09.59.35-04.00 logSource="pkg/controller/backup_controller.go:598"
+time="2022-08-25T13:59:35Z" level=info msg="Setting up backup store to check for backup existence" backup=arthurai/arthur-backup-2022-08-25t09.59.35-04.00 logSource="pkg/controller/backup_controller.go:608"
+time="2022-08-25T13:59:36Z" level=info msg="Writing backup version file" backup=arthurai/arthur-backup-2022-08-25t09.59.35-04.00 logSource="pkg/backup/backup.go:192"
+time="2022-08-25T13:59:36Z" level=info msg="Including namespaces: arthurai" backup=arthurai/arthur-backup-2022-08-25t09.59.35-04.00 logSource="pkg/backup/backup.go:198"
+time="2022-08-25T13:59:36Z" level=info msg="Excluding namespaces: <none>" backup=arthurai/arthur-backup-2022-08-25t09.59.35-04.00 logSource="pkg/backup/backup.go:199"
+time="2022-08-25T13:59:36Z" level=info msg="Including resources: *" backup=arthurai/arthur-backup-2022-08-25t09.59.35-04.00 logSource="pkg/backup/backup.go:202"
+time="2022-08-25T13:59:36Z" level=info msg="Excluding resources: <none>" backup=arthurai/arthur-backup-2022-08-25t09.59.35-04.00 logSource="pkg/backup/backup.go:203"
+time="2022-08-25T13:59:36Z" level=info msg="Backing up all pod volumes using Restic: false" backup=arthurai/arthur-backup-2022-08-25t09.59.35-04.00 logSource="pkg/backup/backup.go:204"
+
+<truncated>
+```
+
+Finally, you can get an overview of the backup using the Velero CLI:
+
+```bash
+$ velero backup describe arthur-backup-2022-08-25t09.59.35-04.00 -n arthurai
+Name:         arthur-backup-2022-08-25t09.59.35-04.00
+Namespace:    arthurai
+Labels:       velero.io/storage-location=default
+Annotations:  velero.io/source-cluster-k8s-gitversion=v1.21.14-eks-18ef993
+              velero.io/source-cluster-k8s-major-version=1
+              velero.io/source-cluster-k8s-minor-version=21+
+
+Phase:  Completed
+
+Errors:    0
+Warnings:  0
+
+Namespaces:
+  Included:  arthurai
+  Excluded:  <none>
+
+Resources:
+  Included:        *
+  Excluded:        <none>
+  Cluster-scoped:  auto
+
+Label selector:  <none>
+
+Storage Location:  default
+
+Velero-Native Snapshot PVs:  auto
+
+TTL:  720h0m0s
+
+Hooks:  <none>
+
+Backup Format Version:  1.1.0
+szExpiration:  2022-09-24 09:59:35 -0400 EDT
+
+Total items to be backed up:  654
+Items backed up:              654
+
+Velero-Native Snapshots:  3 of 3 snapshots completed successfully (specify --details for more information)
+
+Restic Backups (specify --details for more information):
+  Completed:  23
+```
+
+
+### Working with Velero - Restore
+
+Similar to backup, restore happens using the Velero Cli. A restore takes a Backup object and then executes the restore procedure.
+
+You can execute a restore with the following Velero CLI command:
+
+```bash
+$ velero restore create \
+    --from-backup arthur-backup-2022-08-25t09.59.35-04.00 \
+    --namespace arthurai \
+    --restore-volumes=true
+```
+
+Just like with the Backup, Velero will create a Restore Velero Resource, which you can inspect with the Velero CLI:
+
+```bash
+$ velero restore get -n arthurai
+NAME                                                     BACKUP                                    STATUS            STARTED                         COMPLETED                       ERRORS   WARNINGS   CREATED                         SELECTOR
+arthur-backup-2022-08-26t10.37.52-04.00-20220826110520   arthur-backup-2022-08-26t10.37.52-04.00   Completed         2022-08-26 11:06:16 -0400 EDT   2022-08-26 11:08:00 -0400 EDT   0        35         2022-08-26 11:05:21 -0400 EDT   <none>
+```

files/arthur-docs-markdown/platform-management/ongoing-maintenance/index.md.txt

@@ -0,0 +1,31 @@
+# Ongoing Maintenance
+
+This section covers information for platform administrators responsible for ongoing maintenance of the Arthur platform.
+
+## {doc}`audit_log`
+This page shows how to audit log of all calls to sensitive Arthur endpoints.
+
+## {doc}`Administration <post_install>`
+This page shows how handle administration settings for managing your Arthur account.
+
+## {doc}`organizations_and_users`
+This page shows how to manage and create Arthur organizations and 
+users.
+
+## {doc}`backup`
+This page shows how to use Arthur's managed backup and restore capability.
+
+## {doc}`upgrade`
+This page shows how to upgrade Arthur platform components.
+***
+
+```{toctree}
+:maxdepth: 2
+:hidden:
+
+Audit Log <audit_log>
+Administration <post_install>
+Organizations and Users <organizations_and_users>
+Backup and Restore <backup>
+Upgrade <upgrade>
+```

files/arthur-docs-markdown/platform-management/ongoing-maintenance/organizations_and_users.md.txt

@@ -0,0 +1,182 @@
+# Organizations and Users
+
+By default, a new organization, "My Organization" is created by the installer for convenience. You can also create new 
+organizations using the API with the `superadmin` user.
+
+### Log in with superadmin credentials
+
+The first thing you will need is a `superadmin` authorization token. To obtain this, you will need to make a `POST`
+request to your organization's `/login` endpoint with the password set in the Admin Console Config page.
+
+```
+POST /login
+```
+```json
+{
+   "login": "superadmin",
+   "password": "<superadmin-password>"
+}
+```
+
+The response will look like this:
+
+```json
+{
+    "id": "ed1dcb56-352a-4130-8f52-1fd1225196b1",
+    "first_name": "Super",
+    "last_name": "Admin",
+    "email": "superadmin@arthur.ai",
+    "username": "superadmin",
+    "roles": null,
+    "active": true,
+    "show_intro_sequence": true,
+    "help_mode_enabled": false,
+    "created_at": "2021-08-09T19:57:44.92047Z"
+}
+```
+
+The response will also include a `set-cookie` HTTP header with an authorization token.` Copy the authorization token value and use it in subsequent
+requests as your auth token.
+
+```
+set-cookie: Authorization=<authorization-token>; Path=/; Expires=Mon, 30 Aug 2021 16:51:07 GMT; Secure;
+```
+
+#### cURL example
+
+```bash
+curl --location --request POST 'https://<your-domain>/api/v3/login' --header 'Content-Type: application/json' --data-raw '{ "login": "superadmin", "password": "<superadmin-password>" }' -v
+```
+
+### Create a New Organization
+To create a new organization, you will need to make a `POST` request to `/organizations` with the body specifying
+the name. Ensure you are using a super admin authentication token to make this request.
+
+```
+POST /organizations
+```
+```json
+{
+  "name": "my-new-organization"
+}
+```
+
+The response will look like this:
+
+```json
+{
+    "id": "38faff8b-4edf-44c5-b103-aeca4ea71110",
+    "name": "my-new-organization",
+    "plan": "enterprise",
+    "created_at": "2021-08-18T19:51:22.291504554Z"
+}
+```
+
+Remember to save the `id`; you will need this to add users to your organization.
+
+#### cURL Example
+
+```bash
+curl --location --request POST '<your-domain>/api/v3/organizations' --header 'Content-Type: application/json' --header 'Authorization: <your-superadmin-access-control-token>' --data-raw '{ "name": "my-new-organization" }' -v
+```
+
+### Create The First User in an Organization
+
+To create a new user in the new organization, you will need to make a `POST` request to
+`/users?organization_id=<your_organization_id>` using a super admin authentication token. You can set the role of the new user to `Administrator`,
+`Model Owner`, or `User`. Refer to the [authorization page](../access-control-overview/standard_access_control) for the description of the roles.
+
+```
+POST /users?organization_id=<your_organization_id>
+```
+```json
+{
+    "username": "newuser",
+    "email": "newuser@email.com",
+    "password": "G00dP@$$w0rd!",
+    "first_name": "New",
+    "last_name": "User",
+    "roles": [
+        "Administrator"
+    ],
+    "alert_notifications_enabled": true
+}
+```
+
+The response will look like this.
+
+```json
+{
+    "id": "b6554927-9ac4-4531-bf76-fe640b8223b7",
+    "first_name": "New",
+    "last_name": "User",
+    "email": "newuser@email.com",
+    "username": "newuser",
+    "roles": null,
+    "active": true,
+    "show_intro_sequence": true,
+    "help_mode_enabled": true,
+    "created_at": "2021-08-18T20:20:18.535137592Z"
+}
+```
+
+You can now log in to the dashboard as this user.
+
+#### cURL Example
+This action can be performed as either super administrator or organization administrator.
+If you'd like to use an organization administrator, repeat the `/login` API call performed earlier with the credentials for that user and save the returned authorization token. 
+
+```bash
+curl --location --request POST 'https://<your-domain>/api/v3/users?organization_id=<your-organization-id>' --header 'Content-Type: application/json' --header 'Authorization: <your-superadmin-token>' --data-raw '{ "username": "<username>", "email": "<email-address>", "password": "<password>", "first_name": "<first-name>", "last_name": "<last-name>", "roles": [ "Administrator" ], "alert_notifications_enabled": true }'
+```
+
+### Adding Additional Users
+
+Although you can continue to create users through the API, it is generally easier to create an `Administrator` user
+and then invite additional users from the UI. To add additional users this way, log in to Arthur AI with an `Administrator`
+user on a web browser and follow these steps:
+
+1. In the top right corner, you will see a series of icons. Click on the Organization icon that looks like a tree with three nodes.
+
+   ![My Organization Panel](/_static/images/platform/my_organization_panel.png "My Organization Panel")
+
+2. You will see a dropdown menu. Click on `Manage Members`
+3. Under the heading, `Invite Members`, you can type in the email address of the person you wish to invite.
+   That person will receive instructions for how to create a user in the organization via email.
+
+   ![Invite Members Section](/_static/images/platform/invite_members_section.png "Invite Members Section")
+
+4. Once the new user follows the emailed instructions, they will be able to log in with their newly created
+   username and password. You will then be able to view that new user on this `Manage Members` page.
+5. As an `Administrator`, you can continue to use this page to manage users and roles.
+
+
+### Adding Existing Users To Existing Organizations via API:
+To add an existing user to an existing organization create a `PATCH` request to `/organizations/<org_id>/users`. 
+Supplying in the body a json object defining the role (`Administrator`,
+`Model Owner`, or `User`) you want to add the user with. Any attributes other than roles that are supplied in the body 
+will effect the user across all organizations that user is a part of.
+
+```
+PATCH /organizations/<org_id>/users
+```
+```json
+[
+    {
+      "user_id": "b6554927-9ac4-4531-bf76-fe640b8223b7",
+      "role": "Model Owner"
+    } ,
+    {
+      "user_id": "b6554927-9ac4-4531-bf76-fe640b8223b7",
+      "role": "Model Owner"
+    } 
+]
+```
+
+The response will look like this.
+
+```json
+{
+  "updated_user_count": 10
+}
+```

files/arthur-docs-markdown/platform-management/ongoing-maintenance/post_install.md.txt

@@ -0,0 +1,33 @@
+# Post Installation Steps
+
+By default, a new organization, "My Organization" is created by the installer for convenience. You can also create new 
+organizations using the API with the `superadmin` user. Full instructions for creating new users and organizations can
+be found {doc}`here <organizations_and_users>`.
+
+To access the UI for the default organization dashboard, visit `https://your_arthur_domain` from your Web browser. 
+Login with `admin` username and `SuperSecret` password. Make sure to change the password as soon as possible.
+
+To get started with onboarding your models, refer to the [Quickstart](../../user-guide/arthur_quickstart) guide.
+
+## Admin Console
+The Admin Console can be made available via the ingress controller on port 443 by creating a subdomain DNS record that starts with `admin.`
+(e.g. admin.arthur.mydomain.com). For VM install, this eliminates the port 8800 egress requirement on the firewall.
+
+We recommend that you rotate your Admin Console password often. You can reset the password using this command: 
+```shell
+kubectl kots reset-password -n <namespace>
+```
+
+## Update Admin password for Embedded Postgres
+
+The 'postgres' admin user is used to manage the embedded Postgres database. If you would like to update the password for this admin user, you can execute the following commands on the primary database pod:
+```shell
+kubectl exec -it database-master-0 -- psql -U postgres
+Password for user postgres: <type_current_secret>
+psql (11.13)
+Type "help" for help.
+
+postgres=# ALTER ROLE postgres WITH PASSWORD '<insert_new_secret>';
+postgres=# \q
+$ 
+```

files/arthur-docs-markdown/platform-management/ongoing-maintenance/upgrade.md.txt

@@ -0,0 +1,8 @@
+# Upgrading
+
+Arthur's platform installer leverages an open source solution called [Kubernetes Off-The-Shelf (KOTS)](https://github.com/replicatedhq/kots).
+Please refer to the below links for upgrading the platform components.
+
+* [Upgrading the platform application](https://kots.io/kotsadm/updating/updating-kots-apps/)
+* [Upgrading the Admin Console](https://kots.io/kotsadm/updating/updating-admin-console/)
+* [Upgrading the embedded Kubernetes cluster for VM install](https://kots.io/kotsadm/updating/updating-embedded-cluster/)

files/arthur-docs-markdown/platform-management/reference/config_dump.md.txt

@@ -0,0 +1,11 @@
+# Exporting Platform Configurations
+
+After a successful installation of the Arthur platform, you can download the configuration that was used as a `.yaml` file by following the below steps:
+
+1. Navigate to the Admin Console and login.
+![.png image](/_static/images/platform/admin-console-login.png)
+2. Click on the **"View Files"** tab.
+![.png image](/_static/images/platform/admin-console-view-files.png)
+3. Go to **upstream → userdata** and click on `config.yaml` file.
+![.png image](/_static/images/platform/admin-console-config-yaml.png)
+4. Copy the contents of that file and save it for future use (or check-in to source control).

files/arthur-docs-markdown/platform-management/reference/config_template.md.txt

@@ -0,0 +1,371 @@
+# Configuration File Template
+
+The Configuration template for Arthur version `3.2.1` is below:
+
+```yaml
+apiVersion: kots.io/v1beta1
+kind: ConfigValues
+metadata:
+  creationTimestamp: null
+  name: arthur
+spec:
+  values:
+    IAM_permission_type:
+      default: access_keys
+    advanced_cache_options:
+      default: "0"
+    advanced_messaging_connect_cpu_limits:
+      default: "2"
+    advanced_messaging_connect_cpu_limits_not_validate: {}
+    advanced_messaging_connect_cpu_requests:
+      default: "1"
+    advanced_messaging_connect_cpu_requests_not_validate: {}
+    advanced_messaging_connect_heap_options:
+      default: -Xms1g -Xmx3g
+    advanced_messaging_connect_memory_limits:
+      default: 4Gi
+    advanced_messaging_connect_memory_limits_not_validate: {}
+    advanced_messaging_connect_memory_requests:
+      default: 2Gi
+    advanced_messaging_connect_memory_requests_not_validate: {}
+    advanced_olap_options:
+      default: "0"
+    advanced_other:
+      default: "0"
+      value: "1"
+    alert_service_update_rule_metrics:
+      default: "0"
+    api_token_ttl:
+      default: "24"
+    arthur_user_id:
+      default: "1000"
+    audit_log_event_bridge_bus_name: {}
+    audit_log_event_bridge_bus_region: {}
+    audit_log_event_bridge_detail_type:
+      default: events.arthur.ai
+    audit_log_event_bridge_source:
+      default: arthur-audit-log
+    audit_log_sink_destination:
+      default: none
+    batch_workflow_parallelism:
+      default: "120"
+    beta_ui:
+      default: "0"
+    beta_ui_alternate_site:
+      default: "0"
+    beta_ui_hostname: {}
+    bootstrap_job_backoff_limit:
+      default: "100"
+    bootstrap_job_ttl:
+      default: "86400"
+    cache_cpu_limits: {}
+    cache_cpu_limits_not_validate: {}
+    cache_cpu_requests: {}
+    cache_cpu_requests_not_validate: {}
+    cache_memory_limits: {}
+    cache_memory_limits_not_validate: {}
+    cache_memory_requests: {}
+    cache_memory_requests_not_validate: {}
+    cache_password:
+      default: SuperSecret
+      value: VwC3tnE9cpzObSxIhTx9U/34Ky+mA6p8veb9bCk+iqcAEaOarGzGEFf7ozoGxO3m05QY5YTuIx3ezMI694TUX0gj7RHSyHoK
+    cache_replicas:
+      default: "0"
+    cicd_credentials:
+      default: "0"
+    cluster_nodes:
+      # Only Relevant for "fixed" cluster sizes.  Enter the number of nodes in the cluster. This number cannot be decreased from the current value unless it's greater than `6`.
+      default: "1"
+      value: "3"
+    config_job_and_workflow_retention:
+      default: "0"
+    database_admin_password:
+      default: SuperSecret
+      value: VwC3tnE9cpzObSxIhTx9U/34Ky+mA6p8veb9bCk+iqcAEaOarGzGEFf7ozoGxO3m05QY5YTuIx3ezMI694TUX0gj7RHSyHoK
+    database_hostname:
+      # Leave the default configuration to use the embedded database. If you would like to use an external Postgres instance, provide the hostname here and follow this guide: https://docs.arthur.ai/platform-management/installation/externalize_postgres.html.
+      default: database-primary
+    database_password:
+      default: SuperSecret
+      value: VwC3tnE9cpzObSxIhTx9U/34Ky+mA6p8veb9bCk+iqcAEaOarGzGEFf7ozoGxO3m05QY5YTuIx3ezMI694TUX0gj7RHSyHoK
+    database_port:
+      value: "5432"
+    database_ssl_mode:
+      # This option allows you to enable SSL communication between services and the postgres database.  See https://www.postgresql.org/docs/10/libpq-ssl.html for full descriptions of each option.  By default, the postgres database has ssl disabled.
+      default: disable
+    database_username:
+      default: arthurai
+    default_messaging_partition_count:
+      default: "3"
+      value: "1"
+    disable_ssl_redirect_on_ingress:
+      default: "0"
+    email_selection:
+      default: none
+    enable_audit_log:
+      default: "0"
+    enable_olap_backup:
+      default: '"0"'
+    enable_olap_backup_user:
+      default: "0"
+    enable_password_rotation_cache:
+      default: "0"
+    enable_password_rotation_olap:
+      default: "0"
+    existing_database_primary_pvc: {}
+    existing_or_vm:
+      default: existing_cluster
+    fixed_or_autoscale:
+      # The `fixed` mode is recommended for clusters with a fixed number of nodes. The `autoscale` mode is used for clusters that can autoscale and automatically expand their node count.
+      value: fixed
+    full_name_override:
+      default: arthurai
+    global_identity_provider:
+      default: none
+    global_model_limit_Count:
+      default: "500"
+    global_model_limits:
+      default: "0"
+    global_workflow_parallelism:
+      default: "150"
+    http_proxy: {} # Relevant if you are using Explainability and your organization is behind a proxy server.  If PIP and/or Conda need to route through the proxy server to pull down public packages this will set the environment variable HTTP_PROXY to the supplied value. Ex. http://sysproxy.my-company.com:port
+    http_proxy_user: {}
+    https_proxy: {}
+    https_proxy_user: {}
+    ingestion_service_cpu_limits: {}
+    ingestion_service_cpu_limits_not_validate: {}
+    ingestion_service_cpu_requests: {}
+    ingestion_service_cpu_requests_not_validate: {}
+    ingestion_service_memory_limits: {}
+    ingestion_service_memory_limits_not_validate: {}
+    ingestion_service_memory_requests: {}
+    ingestion_service_memory_requests_not_validate: {}
+    ingress_ambassador_enabled:
+      default: "false"
+    ingress_class:
+      default: nginx
+    ingress_hostname:
+      value: arthur.mydomain.ai
+    ingress_namespace_label_key:
+      value: name
+    ingress_namespace_label_value:
+      value: ingress-system
+    ingress_nginx_additional_hostname:
+      value: ""
+    irsa_annotations: {}
+    irsa_annotations_user:
+      default: |
+        eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/my-role
+    k8_storageclass:
+    # Provide Kubernetes StorageClass profile. Use 'gp2' for Amazon EKS, 'default' if you're using embedded Kubernetes provided by the installer
+      value: default
+    kafka_ecosystem_common_replication_calc:
+      default: "1"
+    max_arthur_replicas:
+      default: "1"
+    max_messaging_partition_count:
+      default: "3"
+    max_model_server_replicas:
+      default: "2"
+    messaging_cpu_limit:
+      default: "1"
+    messaging_heap:
+      default: -Xmx2G -Xms1G
+    messaging_memory_limit_and_request:
+      default: 2560Mi
+    messaging_rack_aware_enabled:
+      default: "0"
+    messaging_rack_label:
+      default: topology.kubernetes.io/zone
+    messaging_replicas:
+      default: "3"
+    messaging_sa_create:
+      default: "0"
+    messaging_sa_fullnameoverride: {}
+    messaging_zookeeper_timeout:
+      default: "20000"
+    meta_replicas:
+      default: "0"
+    metric_service_update_default_metrics:
+      default: "0"
+    min_arthur_replicas:
+      default: "1"
+    model_servers_always_on:
+      # For use with what-if and on-demand explainability. See https://docs.arthur.ai/user-guide/explainability.html  If set to "true", then on-demand and what-if explanations are available, but uses additional cluster resources, 1 CPU and 1 GB memory per model with explainability enabled. If set to "false", on-demand and what-if explanations are unavailable, but less cluster usage when there is no data being sent. Regardless of the setting here, streaming explainability will be available if enabled. This only effects what-if and on-demand explanations.
+      default: "true"
+    network_policy_enabled:
+      default: "0"
+    no_proxy: {} # Relevant if you are using Explainability and your organization is behind a proxy server.  If PIP and/or Conda need to route through the proxy server to pull down public packages this will set the environment variable NO_PROXY to the supplied value. Ex. localhost,127.0.0.1,.my-company.com
+    no_proxy_user: {}
+    number_of_olap_backups_to_keep:
+      default: "7"
+    oidc_identity_provider_config_yaml: {}
+    oidc_identity_provider_config_yaml_user: {}
+    olap_backup_s3_bucket:
+      default: arthurai
+    olap_backup_s3_bucket_region:
+      default: us-east-1
+    olap_backup_s3_endpoint:
+      default: s3.us-east-1.amazonaws.com
+    olap_backup_s3_path:
+      default: olap_backups
+    olap_backup_service_account:
+      default: arthurai-arthurai
+    olap_cpu_limits: {}
+    olap_cpu_limits_not_validate: {}
+    olap_cpu_requests:
+      default: 1000m
+    olap_cpu_requests_not_validate: {}
+    olap_database_operator_password:
+      # The OLAP database is installed along with a Kubernetes Operator to manage it.  This operator needs credentials to access the database.  We recommend overwriting the default password below.
+      default: 5ugYLDJ2uLhRdEgz5t
+      value: ch/0gntnboTNbQpxmzx4GuPCRnjqSNwTpOT6FwgQ9q4iY7CHiQLeFQ3snnZgxYnFt4gSyInce3KhYiMR7eebBtGbe5sIuY/aBPAySrSjExfO+1VYPBp176bP+zQ=
+    olap_database_user_password:
+      # Password used internally in our application to query the olap database, currently only supports alpha-numeric characters.
+      default: eQ3iBo8UGh5zqJKQWuEEySrR
+      value: ch/0gntnboTNbQppnGJgGvCjSmPlS/l8orO+UggQ/rstcryCj2r/GRXR8UNr+u3plPIj+uLMdXGGFiRtko6pTsClBoQkoeLXqDVr1jeqsThCZI/bTfovlA==
+    olap_memory_limits: {}
+    olap_memory_limits_not_validate: {}
+    olap_memory_requests:
+      default: 1Gi
+    olap_memory_requests_not_validate: {}
+    olap_node_label_key: {}
+    olap_node_label_value: {}
+    olap_replicas:
+      default: "1"
+    olap_zookeeper_cpu_limits: {}
+    olap_zookeeper_cpu_limits_not_validate: {}
+    olap_zookeeper_cpu_requests:
+      default: 500m
+    olap_zookeeper_cpu_requests_not_validate: {}
+    olap_zookeeper_heap_options:
+      default: -Xms4G -Xmx4G
+    olap_zookeeper_memory_limits: {}
+    olap_zookeeper_memory_limits_not_validate: {}
+    olap_zookeeper_memory_requests:
+      default: 1Gi
+    olap_zookeeper_memory_requests_not_validate: {}
+    password_rotation_cron_schedule:
+      default: 0 0 1 */6 *
+    pending_batch_workflows_limit:
+      default: "100"
+    prometheus_host:
+    # Leave the default configuration if you're using the embedded K8s. Provide your Prometheus hostname if you're running your own K8s.
+      default: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local
+    prometheus_labels:
+      # If your prometheus installation requires labels to identify ServiceMonitors and PrometheusRules, add them here. They should be in yaml format just as you would specify inside the "metadata.labels" block. Do not indent.
+      default: |
+        prometheus: monitor
+        app: prometheus
+    prometheus_namespace:
+      default: monitoring
+    prometheus_port:
+      # Leave the default configuration if you're using the embedded K8s. Provide your Prometheus hostname if you're running your own K8s.
+      default: "9090"
+    pypi_registry_conda: {} # This is set as a channel in the '.condarc' file. Do not include 'https://' prefix (e.g. repository.arthur.ai/repository/conda-proxy/main).
+    pypi_registry_conda_user: {}
+    pypi_registry_index: {} # This maps to the 'index key' in the 'pip.conf' file. Do not include 'https://' prefix (e.g repository.arthur.ai/repository/pypi-virtual/pypi).
+    pypi_registry_index_url: {} # This maps to the 'index-url' key in the 'pip.conf' file. Do not include 'https://' prefix (e.g. repository.arthur.ai/repository/pypi-virtual/simple).
+    pypi_registry_index_url_user: {}
+    pypi_registry_index_user: {}
+    pypi_registry_password:
+      default: bO4Mxhdaevso/029YtUgz98Wk7qPcxEpa1P/uVqG4cy4UY1B3+YN5Q==
+      value: VwC3tnE9cpzObSxIhTx9U/34Ky+mA6p8veb9bCk+iqcAEaOarGzGEFf7ozoGxO3m05QY5YTuIx3ezMI694TUX0gj7RHSyHoK
+    pypi_registry_password_user:
+      value: VwC3tnE9cpzObSxIhTx9U/34Ky+mA6p8veb9bCk+iqcAEaOarGzGEFf7ozoGxO3m05QY5YTuIx3ezMI694TUX0gj7RHSyHoK
+    pypi_registry_username: {}
+    pypi_registry_username_user: {}
+    raw_anaconda_config: {}
+    raw_anaconda_config_user: {}
+    raw_pypi_config: {}
+    raw_pypi_config_user: {}
+    rbac_privileges:
+      # Change to "cluster_scope" to install CRDs too
+      default: namespace_scope
+    run_as_root:
+      default: "0"
+      value: "0"
+    s3_access_key_id:
+      default: access_key
+      value: VwC3tnE9cpzObSxIhTx9U/34Ky+mA6p8veb9bCk+iqcAEaOarGzGEFf7ozoGxO3m05QY5YTuIx3ezMI694TUX0gj7RHSyHoK
+    s3_access_key_id_user:
+      default: access_key
+      value: VwC3tnE9cpzObSxIhTx9U/34Ky+mA6p8veb9bCk+iqcAEaOarGzGEFf7ozoGxO3m05QY5YTuIx3ezMI694TUX0gj7RHSyHoK
+    s3_bucket:
+      default: arthurai
+    s3_bucket_user:
+      default: arthurai
+    s3_region:
+      default: us-east-1
+    s3_region_user:
+      default: us-east-1
+    s3_secret_access_key:
+      default: secret_key
+      value: VwC3tnE9cpzObSxIhTx9U/34Ky+mA6p8veb9bCk+iqcAEaOarGzGEFf7ozoGxO3m05QY5YTuIx3ezMI694TUX0gj7RHSyHoK
+    s3_secret_access_key_user:
+      default: secret_key
+      value: VwC3tnE9cpzObSxIhTx9U/34Ky+mA6p8veb9bCk+iqcAEaOarGzGEFf7ozoGxO3m05QY5YTuIx3ezMI694TUX0gj7RHSyHoK
+    s3_url:
+      default: http://minio:9000
+    s3_url_user:
+      default: http://minio:9000
+    saml_identity_provider_config_yaml: {}
+    saml_identity_provider_config_yaml_user: {}
+    secondary_token_validation_key:
+      value: Aj3ziCI/YcnTT3QR3WAtMNDNEzzqTa8W9iJCoHjNFMteiO6lrcnUKw==
+    ses_region: {}
+    ses_role: {}
+    show_advanced_arthur_microservice_options:
+      default: "0"
+    show_advanced_messaging:
+      default: "0"
+      value: "1"
+    show_hidden_variables:
+      default: "0"
+      value: "0"
+    show_token_signing_and_validation_options:
+      default: "0"
+    signing_cert: {}
+    signing_cert_user: {}
+    signing_private_key: {}
+    signing_private_key_user: {}
+    single_or_ha:
+      # The `single` configuration is a minimal deployment suitable for non-production environments. For production deployment, select `ha`.
+      value: single
+    smtp_from: {} # Provide the email address to send alerts from (e.g. alerts@arthur.ai)
+    smtp_host: {} # Provide the address of the SMTP server (e.g. smtp.arthur.ai)
+    smtp_password:
+      value: VwC3tnE9cpzObSxIhTx9U/34Ky+mA6p8veb9bCk+iqcAEaOarGzGEFf7ozoGxO3m05QY5YTuIx3ezMI694TUX0gj7RHSyHoK
+    smtp_port: {}
+    smtp_user: {}
+    superadmin_email:
+      default: superadmin@myorg.com
+    superadmin_firstname:
+      default: Super
+    superadmin_lastname:
+      default: Admin
+    superadmin_password:
+      default: SuperSecret
+      value: VwC3tnE9cpzObSxIhTx9U/34Ky+mA6p8veb9bCk+iqcAEaOarGzGEFf7ozoGxO3m05QY5YTuIx3ezMI694TUX0gj7RHSyHoK
+    superadmin_username:
+      value: superadmin
+    token_signing_primary_key:
+      value: YSDFzjg5I83KMBJ+wHQmU/ejDQ7tTthIpaDcCRM+iqcDTofiul7DZzTblFkb0e2U0+UJ74TuIx28oGnxPM+pkmKlc1yx2uvj
+    use_external_blob_storage:
+      # Select "Yes" if and only if you are supplying your own S3 compatible storage, otherwise select "No" to use the embedded blob storage.
+      default: "no"
+    use_external_postgres:
+      default: "no"
+    use_raw_python_repository_configs:
+      # The PyPi registry section is only relevant when using the explainability enrichment (https://docs.arthur.ai/user-guide/enrichments.html#explainability). 
+      # Provide your private PyPi registry if you have an airgapped enrivonment or your model requirements file includes packages only hosted in a private repository. 
+      # Leaving this section blank will cause the public PyPi to be used. If the public PyPi is inaccessible from the cluster, the explainability feature will not work.
+      default: "no"
+    use_smtp:
+      default: "0"
+    workflow_ttl_seconds:
+      default: "3600"
+    workflow_ttl_seconds_after_success:
+      default: "60"
+status: {}
+```