{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/dev-37signals-com/2025-01-02-monitoring-10-petabytes-of-data-in-pure-storage/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "As the final part of our move out of the cloud, we are working on [moving 10 petabytes of data out of AWS Simple Storage Service (S3)](https://world.hey.com/dhh/our-cloud-exit-savings-will-now-top-ten-million-over-five-years-c7d9b5bd). After exploring different alternatives, we decided to go with [Pure Storage](https://www.purestorage.com) FlashBlade solution.\n\nWe store different kinds of information on S3, from the attachments customers upload to Basecamp to the Prometheus long-term metrics. On top of that, Pure’s system also provides filesystem-based capabilities, enabling other relevant usages, such as database backup storage. This makes the system a top priority for observability.\n\nAlthough the system has great reliability, out-of-the-box internal alerting, and autonomous ticket creation, it would also be good to have our metrics and alerts to facilitate problem-solving and ensure any disruptions are prioritized and handled.\n\nFor more context on our current Prometheus setup, see [how we use Prometheus at 37signals](https://dev.37signals.com/prometheus-metrics-at-37signals/).\n\n* * *\n\n## Pure OpenMetrics exporter\n\nPure maintains two OpenMetrics exporters, [pure-fb-openmetrics-exporter](https://github.com/PureStorage-OpenConnect/pure-fb-openmetrics-exporter) and [pure-fa-openmetrics-exporter](https://github.com/PureStorage-OpenConnect/pure-fa-openmetrics-exporter). Since we use Pure FlashBlade (fb), this post covers pure-fb-openmetrics-exporter, although overall usage should be similar. The setup is straightforward and requires only binary and basic authentication installation.\n\nHere is a snippet of our Chef recipe that installs it:\n\n```\npure_api_token = \"token\" # If you use Chef, your token should come from an encrypted databag. Changed to hardcoded here to simplify\nPURE_EXPORTER_VERSION = \"1.0.13\".freeze # Generally, we use Chef node metadata for version management. Changed to hardcoded to simplify\n\ndirectory \"/opt/pure_exporter/#{PURE_EXPORTER_VERSION}\" do\n recursive true\n owner 'pure_exporter'\n group 'pure_exporter'\nend\n\n# Avoid recreating under /tmp after reboot if target_binary is already there\ntarget_binary = \"/opt/pure_exporter/#{PURE_EXPORTER_VERSION}/pure-fb-openmetrics-exporter\"\n\nremote_file \"/tmp/pure-fb-openmetrics-exporter-v#{PURE_EXPORTER_VERSION}-linux-amd64.tar.gz\" do\n source \"https://github.com/PureStorage-OpenConnect/pure-fb-openmetrics-exporter/releases/download/v#{PURE_EXPORTER_VERSION}/pure-fb-openmetrics-exporter-v#{PURE_EXPORTER_VERSION}-linux-amd64.tar.gz\"\n not_if { ::File.exist?(target_binary) }\nend\n\narchive_file \"/tmp/pure-fb-openmetrics-exporter-v#{PURE_EXPORTER_VERSION}-linux-amd64.tar.gz\" do\n destination \"/tmp/pure-fb-openmetrics-exporter-v#{PURE_EXPORTER_VERSION}\"\n action :extract\n not_if { ::File.exist?(target_binary) }\nend\n\nexecute \"copy binary\" do\n command \"sudo cp /tmp/pure-fb-openmetrics-exporter-v#{PURE_EXPORTER_VERSION}/pure-fb-openmetrics-exporter /opt/pure_exporter/#{PURE_EXPORTER_VERSION}/pure-exporter\"\n creates \"/opt/pure_exporter/#{PURE_EXPORTER_VERSION}/pure-exporter\"\n not_if { ::File.exist?(target_binary) }\nend\n\ntokens = <:9491']\n labels:\n environment: 'production'\n job: pure_exporter\n params:\n endpoint: [main] # From the tokens configuration above\n```\n\nFor a production-ready setup, we are using a slightly different approach. The exporter supports the usage of specific metric paths to allow for split Prometheus jobs configuration that reduces the overhead of pulling the metrics all at once:\n\n```\n- job_name: pure_exporter_array\n metrics_path: /metrics/array\n static_configs:\n - targets: ['<%= @hostname %>:9491']\n labels:\n environment: 'production'\n job: pure_exporter\n metric_relabel_configs:\n - source_labels: [name]\n target_label: ch\n regex: \"([^.]+).*\"\n replacement: \"$1\"\n action: replace\n - source_labels: [name]\n target_label: fb\n regex: \"[^.]+\\\\.([^.]+).*\"\n replacement: \"$1\"\n action: replace\n - source_labels: [name]\n target_label: bay\n regex: \"[^.]+\\\\.[^.]+\\\\.([^.]+)\"\n replacement: \"$1\"\n action: replace\n params:\n endpoint: [main] # From the tokens configuration above\n- job_name: pure_exporter_clients\n metrics_path: /metrics/clients\n static_configs:\n - targets: ['<%= @hostname %>:9491']\n labels:\n environment: 'production'\n job: pure_exporter\n params:\n endpoint: [main] # From the tokens configuration above\n- job_name: pure_exporter_usage\n metrics_path: /metrics/usage\n static_configs:\n - targets: ['<%= @hostname %>:9491']\n labels:\n environment: 'production'\n job: pure_exporter\n params:\n endpoint: [main]\n- job_name: pure_exporter_policies\n metrics_path: /metrics/policies\n static_configs:\n - targets: ['<%= @hostname %>:9491']\n labels:\n environment: 'production'\n job: pure_exporter\n params:\n endpoint: [main] # From the tokens configuration above\n```\n\nWe also configure some `metric_relabel_configs` to extract labels from `name` using regex. Those labels help reduce the complexity of queries that aggregate metrics by different components.\n\nDetailed documentation on the available metrics can be found [here](https://github.com/PureStorage-OpenConnect/pure-fb-openmetrics-exporter/blob/main/specification/metrics/purefb-metrics.md).\n\n* * *\n\n## Alerts\n\n### Auto Generated Alerts\n\nAs I shared earlier, the system has an internal Alerting module that automatically triggers alerts for critical situations and creates tickets. To cover those alerts on the Prometheus side, we added an alerting configuration of our own that relies on the incoming severities:\n\n```\n- alert: PureAlert\n annotations:\n summary: '{{ $labels.summary }}'\n description: '{{ $labels.component_type }} - {{ $labels.component_name }} - {{ $labels.action }} - {{ $labels.kburl }}'\n dashboard: 'https://grafana/your-dashboard'\n expr: purefb_alerts_open{environment=\"production\"} == 1\n for: 1m\n```\n\nWe still need to evaluate how the pure-generated alerts will interact with the custom alerts I will cover below, and we might decide to stick to one or the other depending on what we find out.\n\n### Hardware\n\nBefore I continue, the image below helps visualize how some of the Pure FlashBlade components are physically organized:\n\nBecause of Pure’s reliability, most isolated hardware failures do not require the immediate attention of an Ops team member. To cover the most basic hardware failures, we configure an alert that sends a message to the Ops Basecamp 4 project chat:\n\n```\n- alert: PureHardwareFailed\n annotations:\n summary: Hardware {{ $labels.name }} in chassis {{ $labels.ch }} is failed\n description: 'The Pure Storage hardware {{ $labels.name }} in chassis {{ $labels.ch }} is failed'\n dashboard: 'https://grafana/your-dashboard'\n expr: purefb_hardware_health == 0\n for: 1m\n labels:\n severity: chat-notification\n```\n\nWe also configure alerts that check for multiple hardware failures of the same type. This doesn’t mean two simultaneous failures will result in a critical state, but it is a fair guardrail for unexpected scenarios. We also expect those situations to be rare, keeping the risk of causing unnecessary noise low.\n\n```\n- alert: PureMultipleHardwareFailed\n annotations:\n summary: Pure chassis {{ $labels.ch }} has {{ $value }} failed {{ $labels.type }}\n description: 'The Pure Storage chassis {{ $labels.ch }} has {{ $value }} failed {{ $labels.type }}, close to the healthy limit of two simultaneous failures. Ensure that the hardware failures are being worked on'\n dashboard: 'https://grafana/your-dashboard'\n expr: count(purefb_hardware_health{type!~\"eth|mgmt_port|bay\"} == 0) by (ch,type,environment) > 1\n for: 1m\n labels:\n severity: page\n # We are looking for multiple failed bays in the same blade\n- alert: PureMultipleBaysFailed\n annotations:\n summary: Pure chassis {{ $labels.ch }} has fb {{ $labels.fb }} with {{ $value }} failed bays\n description: 'The Pure Storage chassis {{ $labels.ch }} has fb {{ $labels.fb }} with {{ $value }} failed bays, close to the healthy limit of two simultaneous failures. Ensure that the hardware failures are being worked on'\n dashboard: 'https://grafana/your-dashboard'\n expr: count(purefb_hardware_health{type=\"bay\"} == 0) by (ch,type,fb,environment) > 1\n for: 1m\n labels:\n severity: page\n```\n\nFinally, we configure high-level alerts for chassis and XFM failures:\n\n```\n- alert: PureChassisFailed\n annotations:\n summary: Chassis {{ $labels.name }} is failed\n description: 'The Pure Storage hardware chassis {{ $labels.name }} is failed'\n dashboard: 'https://grafana/your-dashboard'\n expr: purefb_hardware_health{type=\"ch\"} == 0\n for: 1m\n labels:\n severity: page\n- alert: PureXFMFailed\n annotations:\n summary: Xternal Fabric Module {{ $labels.name }} is failed\n description: 'The Pure Storage hardware Xternal fabric module {{ $labels.name }} is failed'\n dashboard: 'https://grafana/your-dashboard'\n expr: purefb_hardware_health{type=\"xfm\"} == 0\n for: 1m\n labels:\n severity: page\n```\n\n### Latency\n\nUsing the metric `purefb_array_performance_latency_usec` we can set a threshold for all the different protocols and dimensions (read, write, etc), so we are alerted if any problem causes the latency to go above an expected level.\n\n```\n- alert: PureLatencyHigh\n annotations:\n summary: Pure {{ $labels.dimension }} - {{ $labels.protocol }} latency high\n description: 'Pure {{ $labels.protocol }} latency for dimension {{ $labels.dimension }} is above 100ms'\n dashboard: 'https://grafana/your-dashboard'\n expr: (avg_over_time(purefb_array_performance_latency_usec{protocol=\"all\"}[30m]) * 0.001)\n for: 1m\n labels:\n severity: chat-notification\n```\n\n### Saturation\n\nFor saturation, we are primarily worried about something unexpected causing excessive use of array space, increasing the risk of hitting the cluster capacity. With that in mind, it’s good to have a simple alert in place, even if we don’t expect it to fire anytime soon:\n\n```\n- alert: PureArraySpace\n annotations:\n summary: Pure Cluster {{ $labels.instance }} available space is expected to be below 10%\n description: 'The array space for pure cluster {{ $labels.instance }} is expected to be below 10% in a month, please investigate and ensure there is no risk of running out of capacity'\n dashboard: 'https://grafana/your-dashboard'\n expr: (predict_linear(purefb_array_space_bytes{space=\"empty\",type=\"array\"}[30d], 730 * 3600)) < (purefb_array_space_bytes{space=\"capacity\",type=\"array\"} * 0.10)\n for: 1m\n labels:\n severity: chat-notification\n```\n\n### HTTP\n\nWe use BigIp load balancers to front-end the cluster, which means that all the alerts we already had in place for the BigIp HTTP profiles, virtual servers, and pools also cover access to Pure. The solution for each organization on this topic will be different, but it is a good practice to keep an eye on HTTP status codes and throughput.\n\n* * *\n\n## Grafana Dashboards\n\nThe project’s GitHub repository includes [JSON files for Grafana dashboards](https://github.com/PureStorage-OpenConnect/pure-fb-openmetrics-exporter/tree/main/extra/grafana) that are based on the metrics generated by the exporter. With simple adjustments to fit each setup, it’s possible to import them quickly.\n\n* * *\n\n## Wrapping up\n\nOn top of the system’s built-in capabilities, Pure also provides options to integrate their system into well-known tools like Prometheus and Grafana, facilitating the process of managing the cluster the same way we manage everything else. I hope this post helps any other team interested in working with them better understand the effort involved.\n\nThanks for reading!\n\n* * *\n\n- [Other posts in the “On-prem observability” series](https://dev.37signals.com/series/on-prem-observability/)\n", "authors": [ "Victor Bogo" ], "captured_at": "2026-09-19T21:30:23.395188495+00:00", "category": "programming", "content_html": "

As the final part of our move out of the cloud, we are working on moving 10 petabytes of data out of AWS Simple Storage Service (S3). After exploring different alternatives, we decided to go with Pure Storage FlashBlade solution.

\n

We store different kinds of information on S3, from the attachments customers upload to Basecamp to the Prometheus long-term metrics. On top of that, Pure’s system also provides filesystem-based capabilities, enabling other relevant usages, such as database backup storage. This makes the system a top priority for observability.

\n

Although the system has great reliability, out-of-the-box internal alerting, and autonomous ticket creation, it would also be good to have our metrics and alerts to facilitate problem-solving and ensure any disruptions are prioritized and handled.

\n

For more context on our current Prometheus setup, see how we use Prometheus at 37signals.

\n
\n

Pure OpenMetrics exporter

\n

Pure maintains two OpenMetrics exporters, pure-fb-openmetrics-exporter and pure-fa-openmetrics-exporter. Since we use Pure FlashBlade (fb), this post covers pure-fb-openmetrics-exporter, although overall usage should be similar. The setup is straightforward and requires only binary and basic authentication installation.

\n

Here is a snippet of our Chef recipe that installs it:

\n
pure_api_token = "token" # If you use Chef, your token should come from an encrypted databag. Changed to hardcoded here to simplify\nPURE_EXPORTER_VERSION = "1.0.13".freeze # Generally, we use Chef node metadata for version management. Changed to hardcoded to simplify\n\ndirectory "/opt/pure_exporter/#{PURE_EXPORTER_VERSION}" do\n recursive true\n owner 'pure_exporter'\n group 'pure_exporter'\nend\n\n# Avoid recreating under /tmp after reboot if target_binary is already there\ntarget_binary = "/opt/pure_exporter/#{PURE_EXPORTER_VERSION}/pure-fb-openmetrics-exporter"\n\nremote_file "/tmp/pure-fb-openmetrics-exporter-v#{PURE_EXPORTER_VERSION}-linux-amd64.tar.gz" do\n source "https://github.com/PureStorage-OpenConnect/pure-fb-openmetrics-exporter/releases/download/v#{PURE_EXPORTER_VERSION}/pure-fb-openmetrics-exporter-v#{PURE_EXPORTER_VERSION}-linux-amd64.tar.gz"\n not_if { ::File.exist?(target_binary) }\nend\n\narchive_file "/tmp/pure-fb-openmetrics-exporter-v#{PURE_EXPORTER_VERSION}-linux-amd64.tar.gz" do\n destination "/tmp/pure-fb-openmetrics-exporter-v#{PURE_EXPORTER_VERSION}"\n action :extract\n not_if { ::File.exist?(target_binary) }\nend\n\nexecute "copy binary" do\n command "sudo cp /tmp/pure-fb-openmetrics-exporter-v#{PURE_EXPORTER_VERSION}/pure-fb-openmetrics-exporter /opt/pure_exporter/#{PURE_EXPORTER_VERSION}/pure-exporter"\n creates "/opt/pure_exporter/#{PURE_EXPORTER_VERSION}/pure-exporter"\n not_if { ::File.exist?(target_binary) }\nend\n\ntokens = <<EOF\nmain:\n address: purestorage-mgmt.mydomain.com\n api_token: #{pure_api_token['token']}\nEOF\n\nfile "/opt/pure_exporter/tokens.yml" do\n content tokens\n owner 'pure_exporter'\n group 'pure_exporter'\n sensitive true\nend\n\nsystemd_unit 'pure-exporter.service' do\n content <<-EOU\n # Caution: Chef managed content. This is a file resource from #{cookbook_name}::#{recipe_name}\n #\n\n [Unit]\n Description=Pure Exporter\n After=network.target\n\n [Service]\n Restart=on-failure\n PIDFile=/var/run/pure-exporter.pid\n User=pure_exporter\n Group=pure_exporter\n ExecStart=/opt/pure_exporter/#{PURE_EXPORTER_VERSION}/pure-exporter \\\n --tokens=/opt/pure_exporter/tokens.yml\n\n ExecReload=/bin/kill -HUP $MAINPID\n SyslogIdentifier=pure-exporter\n\n [Install]\n WantedBy=multi-user.target\n EOU\n\n action [ :create, :enable, :start ]\n notifies :reload, "service[pure-exporter]"\nend\n\nservice 'pure-exporter'\n
\n
\n

Prometheus Job Configuration

\n

The simplest way of ingesting the metrics is to configure a basic Job without any customization:

\n
- job_name: pure_exporter\n  metrics_path: /metrics\n  static_configs:\n  - targets: ['<%= @hostname %>:9491']\n    labels:\n      environment: 'production'\n      job: pure_exporter\n  params:\n    endpoint: [main] # From the tokens configuration above\n
\n

For a production-ready setup, we are using a slightly different approach. The exporter supports the usage of specific metric paths to allow for split Prometheus jobs configuration that reduces the overhead of pulling the metrics all at once:

\n
- job_name: pure_exporter_array\n  metrics_path: /metrics/array\n  static_configs:\n  - targets: ['<%= @hostname %>:9491']\n    labels:\n      environment: 'production'\n      job: pure_exporter\n  metric_relabel_configs:\n  - source_labels: [name]\n    target_label: ch\n    regex: "([^.]+).*"\n    replacement: "$1"\n    action: replace\n  - source_labels: [name]\n    target_label: fb\n    regex: "[^.]+\\\\.([^.]+).*"\n    replacement: "$1"\n    action: replace\n  - source_labels: [name]\n    target_label: bay\n    regex: "[^.]+\\\\.[^.]+\\\\.([^.]+)"\n    replacement: "$1"\n    action: replace\n  params:\n    endpoint: [main] # From the tokens configuration above\n- job_name: pure_exporter_clients\n  metrics_path: /metrics/clients\n  static_configs:\n  - targets: ['<%= @hostname %>:9491']\n    labels:\n      environment: 'production'\n      job: pure_exporter\n  params:\n    endpoint: [main] # From the tokens configuration above\n- job_name: pure_exporter_usage\n  metrics_path: /metrics/usage\n  static_configs:\n  - targets: ['<%= @hostname %>:9491']\n    labels:\n      environment: 'production'\n      job: pure_exporter\n  params:\n    endpoint: [main]\n- job_name: pure_exporter_policies\n  metrics_path: /metrics/policies\n  static_configs:\n  - targets: ['<%= @hostname %>:9491']\n    labels:\n      environment: 'production'\n      job: pure_exporter\n  params:\n    endpoint: [main] # From the tokens configuration above\n
\n

We also configure some metric_relabel_configs to extract labels from name using regex. Those labels help reduce the complexity of queries that aggregate metrics by different components.

\n

Detailed documentation on the available metrics can be found here.

\n
\n

Alerts

\n

Auto Generated Alerts

\n

As I shared earlier, the system has an internal Alerting module that automatically triggers alerts for critical situations and creates tickets. To cover those alerts on the Prometheus side, we added an alerting configuration of our own that relies on the incoming severities:

\n
- alert: PureAlert\n  annotations:\n    summary: '{{ $labels.summary }}'\n    description: '{{ $labels.component_type }} - {{ $labels.component_name }} - {{ $labels.action }} - {{ $labels.kburl }}'\n    dashboard: 'https://grafana/your-dashboard'\n  expr: purefb_alerts_open{environment="production"} == 1\n  for: 1m\n
\n

We still need to evaluate how the pure-generated alerts will interact with the custom alerts I will cover below, and we might decide to stick to one or the other depending on what we find out.

\n

Hardware

\n

Before I continue, the image below helps visualize how some of the Pure FlashBlade components are physically organized:

\n

Because of Pure’s reliability, most isolated hardware failures do not require the immediate attention of an Ops team member. To cover the most basic hardware failures, we configure an alert that sends a message to the Ops Basecamp 4 project chat:

\n
- alert: PureHardwareFailed\n  annotations:\n    summary: Hardware {{ $labels.name }} in chassis {{ $labels.ch }} is failed\n    description: 'The Pure Storage hardware {{ $labels.name }} in chassis {{ $labels.ch }} is failed'\n    dashboard: 'https://grafana/your-dashboard'\n  expr: purefb_hardware_health == 0\n  for: 1m\n  labels:\n    severity: chat-notification\n
\n

We also configure alerts that check for multiple hardware failures of the same type. This doesn’t mean two simultaneous failures will result in a critical state, but it is a fair guardrail for unexpected scenarios. We also expect those situations to be rare, keeping the risk of causing unnecessary noise low.

\n
- alert: PureMultipleHardwareFailed\n  annotations:\n    summary: Pure chassis {{ $labels.ch }} has {{ $value }} failed {{ $labels.type }}\n    description: 'The Pure Storage chassis {{ $labels.ch }} has {{ $value }} failed {{ $labels.type }}, close to the healthy limit of two simultaneous failures. Ensure that the hardware failures are being worked on'\n    dashboard: 'https://grafana/your-dashboard'\n  expr: count(purefb_hardware_health{type!~"eth|mgmt_port|bay"} == 0) by (ch,type,environment) > 1\n  for: 1m\n  labels:\n    severity: page\n # We are looking for multiple failed bays in the same blade\n- alert: PureMultipleBaysFailed\n  annotations:\n    summary: Pure chassis {{ $labels.ch }} has fb {{ $labels.fb }} with {{ $value }} failed bays\n    description: 'The Pure Storage chassis {{ $labels.ch }} has fb {{ $labels.fb }} with {{ $value }} failed bays, close to the healthy limit of two simultaneous failures. Ensure that the hardware failures are being worked on'\n    dashboard: 'https://grafana/your-dashboard'\n  expr: count(purefb_hardware_health{type="bay"} == 0) by (ch,type,fb,environment) > 1\n  for: 1m\n  labels:\n    severity: page\n
\n

Finally, we configure high-level alerts for chassis and XFM failures:

\n
- alert: PureChassisFailed\n  annotations:\n    summary: Chassis {{ $labels.name }} is failed\n    description: 'The Pure Storage hardware chassis {{ $labels.name }} is failed'\n    dashboard: 'https://grafana/your-dashboard'\n  expr: purefb_hardware_health{type="ch"} == 0\n  for: 1m\n  labels:\n    severity: page\n- alert: PureXFMFailed\n  annotations:\n    summary: Xternal Fabric Module {{ $labels.name }} is failed\n    description: 'The Pure Storage hardware Xternal fabric module {{ $labels.name }} is failed'\n    dashboard: 'https://grafana/your-dashboard'\n  expr: purefb_hardware_health{type="xfm"} == 0\n  for: 1m\n  labels:\n  severity: page\n
\n

Latency

\n

Using the metric purefb_array_performance_latency_usec we can set a threshold for all the different protocols and dimensions (read, write, etc), so we are alerted if any problem causes the latency to go above an expected level.

\n
- alert: PureLatencyHigh\n  annotations:\n    summary: Pure {{ $labels.dimension }} - {{ $labels.protocol }} latency high\n    description: 'Pure {{ $labels.protocol }} latency for dimension {{ $labels.dimension }} is above 100ms'\n    dashboard: 'https://grafana/your-dashboard'\n  expr: (avg_over_time(purefb_array_performance_latency_usec{protocol="all"}[30m]) * 0.001)\n  for: 1m\n  labels:\n    severity: chat-notification\n
\n

Saturation

\n

For saturation, we are primarily worried about something unexpected causing excessive use of array space, increasing the risk of hitting the cluster capacity. With that in mind, it’s good to have a simple alert in place, even if we don’t expect it to fire anytime soon:

\n
- alert: PureArraySpace\n  annotations:\n    summary: Pure Cluster {{ $labels.instance }} available space is expected to be below 10%\n    description: 'The array space for pure cluster {{ $labels.instance }} is expected to be below 10% in a month, please investigate and ensure there is no risk of running out of capacity'\n    dashboard: 'https://grafana/your-dashboard'\n  expr: (predict_linear(purefb_array_space_bytes{space="empty",type="array"}[30d], 730 * 3600)) < (purefb_array_space_bytes{space="capacity",type="array"} * 0.10)\n  for: 1m\n  labels:\n    severity: chat-notification\n
\n

HTTP

\n

We use BigIp load balancers to front-end the cluster, which means that all the alerts we already had in place for the BigIp HTTP profiles, virtual servers, and pools also cover access to Pure. The solution for each organization on this topic will be different, but it is a good practice to keep an eye on HTTP status codes and throughput.

\n
\n

Grafana Dashboards

\n

The project’s GitHub repository includes JSON files for Grafana dashboards that are based on the metrics generated by the exporter. With simple adjustments to fit each setup, it’s possible to import them quickly.

\n
\n

Wrapping up

\n

On top of the system’s built-in capabilities, Pure also provides options to integrate their system into well-known tools like Prometheus and Grafana, facilitating the process of managing the cluster the same way we manage everything else. I hope this post helps any other team interested in working with them better understand the effort involved.

\n

Thanks for reading!

\n
\n\n", "content_markdown": "As the final part of our move out of the cloud, we are working on [moving 10 petabytes of data out of AWS Simple Storage Service (S3)](https://world.hey.com/dhh/our-cloud-exit-savings-will-now-top-ten-million-over-five-years-c7d9b5bd). After exploring different alternatives, we decided to go with [Pure Storage](https://www.purestorage.com) FlashBlade solution.\n\nWe store different kinds of information on S3, from the attachments customers upload to Basecamp to the Prometheus long-term metrics. On top of that, Pure’s system also provides filesystem-based capabilities, enabling other relevant usages, such as database backup storage. This makes the system a top priority for observability.\n\nAlthough the system has great reliability, out-of-the-box internal alerting, and autonomous ticket creation, it would also be good to have our metrics and alerts to facilitate problem-solving and ensure any disruptions are prioritized and handled.\n\nFor more context on our current Prometheus setup, see [how we use Prometheus at 37signals](https://dev.37signals.com/prometheus-metrics-at-37signals/).\n\n* * *\n\n## Pure OpenMetrics exporter\n\nPure maintains two OpenMetrics exporters, [pure-fb-openmetrics-exporter](https://github.com/PureStorage-OpenConnect/pure-fb-openmetrics-exporter) and [pure-fa-openmetrics-exporter](https://github.com/PureStorage-OpenConnect/pure-fa-openmetrics-exporter). Since we use Pure FlashBlade (fb), this post covers pure-fb-openmetrics-exporter, although overall usage should be similar. The setup is straightforward and requires only binary and basic authentication installation.\n\nHere is a snippet of our Chef recipe that installs it:\n\n```\npure_api_token = \"token\" # If you use Chef, your token should come from an encrypted databag. Changed to hardcoded here to simplify\nPURE_EXPORTER_VERSION = \"1.0.13\".freeze # Generally, we use Chef node metadata for version management. Changed to hardcoded to simplify\n\ndirectory \"/opt/pure_exporter/#{PURE_EXPORTER_VERSION}\" do\n recursive true\n owner 'pure_exporter'\n group 'pure_exporter'\nend\n\n# Avoid recreating under /tmp after reboot if target_binary is already there\ntarget_binary = \"/opt/pure_exporter/#{PURE_EXPORTER_VERSION}/pure-fb-openmetrics-exporter\"\n\nremote_file \"/tmp/pure-fb-openmetrics-exporter-v#{PURE_EXPORTER_VERSION}-linux-amd64.tar.gz\" do\n source \"https://github.com/PureStorage-OpenConnect/pure-fb-openmetrics-exporter/releases/download/v#{PURE_EXPORTER_VERSION}/pure-fb-openmetrics-exporter-v#{PURE_EXPORTER_VERSION}-linux-amd64.tar.gz\"\n not_if { ::File.exist?(target_binary) }\nend\n\narchive_file \"/tmp/pure-fb-openmetrics-exporter-v#{PURE_EXPORTER_VERSION}-linux-amd64.tar.gz\" do\n destination \"/tmp/pure-fb-openmetrics-exporter-v#{PURE_EXPORTER_VERSION}\"\n action :extract\n not_if { ::File.exist?(target_binary) }\nend\n\nexecute \"copy binary\" do\n command \"sudo cp /tmp/pure-fb-openmetrics-exporter-v#{PURE_EXPORTER_VERSION}/pure-fb-openmetrics-exporter /opt/pure_exporter/#{PURE_EXPORTER_VERSION}/pure-exporter\"\n creates \"/opt/pure_exporter/#{PURE_EXPORTER_VERSION}/pure-exporter\"\n not_if { ::File.exist?(target_binary) }\nend\n\ntokens = <:9491']\n labels:\n environment: 'production'\n job: pure_exporter\n params:\n endpoint: [main] # From the tokens configuration above\n```\n\nFor a production-ready setup, we are using a slightly different approach. The exporter supports the usage of specific metric paths to allow for split Prometheus jobs configuration that reduces the overhead of pulling the metrics all at once:\n\n```\n- job_name: pure_exporter_array\n metrics_path: /metrics/array\n static_configs:\n - targets: ['<%= @hostname %>:9491']\n labels:\n environment: 'production'\n job: pure_exporter\n metric_relabel_configs:\n - source_labels: [name]\n target_label: ch\n regex: \"([^.]+).*\"\n replacement: \"$1\"\n action: replace\n - source_labels: [name]\n target_label: fb\n regex: \"[^.]+\\\\.([^.]+).*\"\n replacement: \"$1\"\n action: replace\n - source_labels: [name]\n target_label: bay\n regex: \"[^.]+\\\\.[^.]+\\\\.([^.]+)\"\n replacement: \"$1\"\n action: replace\n params:\n endpoint: [main] # From the tokens configuration above\n- job_name: pure_exporter_clients\n metrics_path: /metrics/clients\n static_configs:\n - targets: ['<%= @hostname %>:9491']\n labels:\n environment: 'production'\n job: pure_exporter\n params:\n endpoint: [main] # From the tokens configuration above\n- job_name: pure_exporter_usage\n metrics_path: /metrics/usage\n static_configs:\n - targets: ['<%= @hostname %>:9491']\n labels:\n environment: 'production'\n job: pure_exporter\n params:\n endpoint: [main]\n- job_name: pure_exporter_policies\n metrics_path: /metrics/policies\n static_configs:\n - targets: ['<%= @hostname %>:9491']\n labels:\n environment: 'production'\n job: pure_exporter\n params:\n endpoint: [main] # From the tokens configuration above\n```\n\nWe also configure some `metric_relabel_configs` to extract labels from `name` using regex. Those labels help reduce the complexity of queries that aggregate metrics by different components.\n\nDetailed documentation on the available metrics can be found [here](https://github.com/PureStorage-OpenConnect/pure-fb-openmetrics-exporter/blob/main/specification/metrics/purefb-metrics.md).\n\n* * *\n\n## Alerts\n\n### Auto Generated Alerts\n\nAs I shared earlier, the system has an internal Alerting module that automatically triggers alerts for critical situations and creates tickets. To cover those alerts on the Prometheus side, we added an alerting configuration of our own that relies on the incoming severities:\n\n```\n- alert: PureAlert\n annotations:\n summary: '{{ $labels.summary }}'\n description: '{{ $labels.component_type }} - {{ $labels.component_name }} - {{ $labels.action }} - {{ $labels.kburl }}'\n dashboard: 'https://grafana/your-dashboard'\n expr: purefb_alerts_open{environment=\"production\"} == 1\n for: 1m\n```\n\nWe still need to evaluate how the pure-generated alerts will interact with the custom alerts I will cover below, and we might decide to stick to one or the other depending on what we find out.\n\n### Hardware\n\nBefore I continue, the image below helps visualize how some of the Pure FlashBlade components are physically organized:\n\nBecause of Pure’s reliability, most isolated hardware failures do not require the immediate attention of an Ops team member. To cover the most basic hardware failures, we configure an alert that sends a message to the Ops Basecamp 4 project chat:\n\n```\n- alert: PureHardwareFailed\n annotations:\n summary: Hardware {{ $labels.name }} in chassis {{ $labels.ch }} is failed\n description: 'The Pure Storage hardware {{ $labels.name }} in chassis {{ $labels.ch }} is failed'\n dashboard: 'https://grafana/your-dashboard'\n expr: purefb_hardware_health == 0\n for: 1m\n labels:\n severity: chat-notification\n```\n\nWe also configure alerts that check for multiple hardware failures of the same type. This doesn’t mean two simultaneous failures will result in a critical state, but it is a fair guardrail for unexpected scenarios. We also expect those situations to be rare, keeping the risk of causing unnecessary noise low.\n\n```\n- alert: PureMultipleHardwareFailed\n annotations:\n summary: Pure chassis {{ $labels.ch }} has {{ $value }} failed {{ $labels.type }}\n description: 'The Pure Storage chassis {{ $labels.ch }} has {{ $value }} failed {{ $labels.type }}, close to the healthy limit of two simultaneous failures. Ensure that the hardware failures are being worked on'\n dashboard: 'https://grafana/your-dashboard'\n expr: count(purefb_hardware_health{type!~\"eth|mgmt_port|bay\"} == 0) by (ch,type,environment) > 1\n for: 1m\n labels:\n severity: page\n # We are looking for multiple failed bays in the same blade\n- alert: PureMultipleBaysFailed\n annotations:\n summary: Pure chassis {{ $labels.ch }} has fb {{ $labels.fb }} with {{ $value }} failed bays\n description: 'The Pure Storage chassis {{ $labels.ch }} has fb {{ $labels.fb }} with {{ $value }} failed bays, close to the healthy limit of two simultaneous failures. Ensure that the hardware failures are being worked on'\n dashboard: 'https://grafana/your-dashboard'\n expr: count(purefb_hardware_health{type=\"bay\"} == 0) by (ch,type,fb,environment) > 1\n for: 1m\n labels:\n severity: page\n```\n\nFinally, we configure high-level alerts for chassis and XFM failures:\n\n```\n- alert: PureChassisFailed\n annotations:\n summary: Chassis {{ $labels.name }} is failed\n description: 'The Pure Storage hardware chassis {{ $labels.name }} is failed'\n dashboard: 'https://grafana/your-dashboard'\n expr: purefb_hardware_health{type=\"ch\"} == 0\n for: 1m\n labels:\n severity: page\n- alert: PureXFMFailed\n annotations:\n summary: Xternal Fabric Module {{ $labels.name }} is failed\n description: 'The Pure Storage hardware Xternal fabric module {{ $labels.name }} is failed'\n dashboard: 'https://grafana/your-dashboard'\n expr: purefb_hardware_health{type=\"xfm\"} == 0\n for: 1m\n labels:\n severity: page\n```\n\n### Latency\n\nUsing the metric `purefb_array_performance_latency_usec` we can set a threshold for all the different protocols and dimensions (read, write, etc), so we are alerted if any problem causes the latency to go above an expected level.\n\n```\n- alert: PureLatencyHigh\n annotations:\n summary: Pure {{ $labels.dimension }} - {{ $labels.protocol }} latency high\n description: 'Pure {{ $labels.protocol }} latency for dimension {{ $labels.dimension }} is above 100ms'\n dashboard: 'https://grafana/your-dashboard'\n expr: (avg_over_time(purefb_array_performance_latency_usec{protocol=\"all\"}[30m]) * 0.001)\n for: 1m\n labels:\n severity: chat-notification\n```\n\n### Saturation\n\nFor saturation, we are primarily worried about something unexpected causing excessive use of array space, increasing the risk of hitting the cluster capacity. With that in mind, it’s good to have a simple alert in place, even if we don’t expect it to fire anytime soon:\n\n```\n- alert: PureArraySpace\n annotations:\n summary: Pure Cluster {{ $labels.instance }} available space is expected to be below 10%\n description: 'The array space for pure cluster {{ $labels.instance }} is expected to be below 10% in a month, please investigate and ensure there is no risk of running out of capacity'\n dashboard: 'https://grafana/your-dashboard'\n expr: (predict_linear(purefb_array_space_bytes{space=\"empty\",type=\"array\"}[30d], 730 * 3600)) < (purefb_array_space_bytes{space=\"capacity\",type=\"array\"} * 0.10)\n for: 1m\n labels:\n severity: chat-notification\n```\n\n### HTTP\n\nWe use BigIp load balancers to front-end the cluster, which means that all the alerts we already had in place for the BigIp HTTP profiles, virtual servers, and pools also cover access to Pure. The solution for each organization on this topic will be different, but it is a good practice to keep an eye on HTTP status codes and throughput.\n\n* * *\n\n## Grafana Dashboards\n\nThe project’s GitHub repository includes [JSON files for Grafana dashboards](https://github.com/PureStorage-OpenConnect/pure-fb-openmetrics-exporter/tree/main/extra/grafana) that are based on the metrics generated by the exporter. With simple adjustments to fit each setup, it’s possible to import them quickly.\n\n* * *\n\n## Wrapping up\n\nOn top of the system’s built-in capabilities, Pure also provides options to integrate their system into well-known tools like Prometheus and Grafana, facilitating the process of managing the cluster the same way we manage everything else. I hope this post helps any other team interested in working with them better understand the effort involved.\n\nThanks for reading!\n\n* * *\n\n- [Other posts in the “On-prem observability” series](https://dev.37signals.com/series/on-prem-observability/)\n", "dateCreated": "2026-09-19T21:30:23.395188495+00:00", "date_created": "2025-01-02T18:00:00+00:00", "date_modified": "2025-01-02T18:00:00+00:00", "date_published": "2025-01-02T18:00:00+00:00", "description": "How we use Prometheus to have metrics and alerts for Pure Storage.", "external_url": "https://dev.37signals.com/pure-storage-monitoring/", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/dev-37signals-com/2025/01/2025-01-02-monitoring-10-petabytes-of-data-in-pure-storage.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/dev-37signals-com/2025/01/2025-01-02-monitoring-10-petabytes-of-data-in-pure-storage.md" }, "id": "urn:aggr:item:ba9eae0f3598c3bf102259d646454b43a8c3a869", "isBasedOn": { "@id": "https://dev.37signals.com/pure-storage-monitoring/" }, "mainEntity": { "@id": "https://dev.37signals.com/pure-storage-monitoring/", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/dev-37signals-com/2025-01-02-monitoring-10-petabytes-of-data-in-pure-storage/#webpage" }, "author": [ { "@type": "Person", "name": "Victor Bogo" } ], "dateModified": "2025-01-02T18:00:00+00:00", "datePublished": "2025-01-02T18:00:00+00:00", "headline": "Monitoring 10 Petabytes of data in Pure Storage", "timeRequired": "PT7M", "url": "https://dev.37signals.com/pure-storage-monitoring/", "wordCount": 1522 }, "name": "Archived snapshot: Monitoring 10 Petabytes of data in Pure Storage", "preview": null, "reading_minutes": 7, "replicated_at": null, "source": "dev-37signals-com", "source_name": "37signals Dev", "summary": "How we use Prometheus to have metrics and alerts for Pure Storage.", "tags": [], "temporalCoverage": "2026-09-19T21:30:23.395188495+00:00", "title": "Monitoring 10 Petabytes of data in Pure Storage", "url": "https://aggr.aymericbeaumet.com/items/dev-37signals-com/2025-01-02-monitoring-10-petabytes-of-data-in-pure-storage/", "word_count": 1522 }