Description A submitted topology carries two lists of blobstore keys, `dependency_jars` and `dependency_artifacts`, which the client fills

Description

A submitted topology carries two lists of blobstore keys, dependency_jars and dependency_artifacts,
which the client fills in after uploading the corresponding blobs. Nimbus performed no validation of their
contents on the submission path, yet acts on them in two places.

During cleanup of a finished topology, Nimbus deletes the keys named in those lists, and the deletion is
performed as the Nimbus subject, for which the blobstore short-circuits its ACL check. A submitter who
listed a key belonging to another topology, such as its -stormjar.jar, could therefore cause
that blob to be deleted when their own topology was cleaned up.

Separately, on acquiring leadership a Nimbus compares the dependency keys of all active topologies against
the blobstore contents and surrenders leadership if any is missing. A single key that does not exist, on a
single active topology, therefore causes every Nimbus to acquire leadership, surrender it and requeue
indefinitely, leaving the cluster without a leader and unable to schedule, clean up or accept submissions.

Mitigation

Upgrade to 3.1.0, where a submission is refused unless every entry in both lists is a dependency blob key
and exists in the blobstore.

Note that this validates new submissions only; a topology stored by an affected version with an invalid list
is unaffected by the upgrade. An operator whose cluster is failing to retain a leader should inspect the
Nimbus log for the dependency keys reported as missing and remove or resubmit the topology naming them.

Users who cannot upgrade immediately should restrict topology submission to trusted principals.

Credit

This issue was discovered by rzo1 while investigating an unrelated blobstore defect.