yogsototh/deft
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
deft/Cisco.org.gpg_archive
Yann Esposito (Yogsototh) a899449835
Archiving daily meeings notes
2019-03-15 09:50:07 +01:00

2947 lines
87 KiB
Org Mode
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# -*- mode: org -*-
Archived entries from file /Users/yaesposi/.deft/Cisco.org.gpg
* [#A] OAuth2 Provider
:PROPERTIES:
:ARCHIVE_TIME: 2019-01-09 Wed 10:52
:ARCHIVE_FILE: ~/.deft/Cisco.org.gpg
:ARCHIVE_OLPATH: Epics
:ARCHIVE_CATEGORY: Cisco.org
:END:
:SCHEDULE: <2018-01-11 Thu>
:END:
** Workflow
Authorize, Approve, Refuse, Token
Threatgrid portal would like access to the (enrich API, response API, ...
Private CTIA Public CTIA) -> scopes
users ses yes, he approves, POST approve, get back auth code, POST that to token
*** =/authorize=
Ask the user to authorize an App to access sub services API:
Grants: https://alexbilbie.com/guide-to-oauth-2-grants/
We'll never support grant type password or client_credentials.
We'll support: =authorization_code= and =refresh_token=.
**** Scopes
Format:
- =<high-level-api>(:<access-type>)?=
- =<high-level-api>(/subset)*(:<access-type>)?=
Examples:
- openid
- email
- collect
- collect:read
- iroh-int/observe (read)
- enrich:write
- global-intel
Meaningful:
- deliberations
- enrichment
- private-intel (read/write)
- global-intel (read/write)
- references (read)
- response (write)
- deliberate, describe, observe
.
*** =/approve=
The User can approve access to an App.
The App get back a Auth Code.
*** =/refuse=
The User can refuse or revoke access to an App.
*** =/token=
The App can ask to refresh tokens.
Tokens returned must be "short" lifetime JWT.
** OAuth2 in IROH-Auth Spec RFC second pass
*** Vocabulary
- Roles
+ resource owner: has the ability to grant access to a protected resource
+ resource server: host of the protected resources
+ client: app making requests to resources
+ authorization server: server issuing access token to the client
- Authorization Grant: creds representing resource owner authorization
- Authorization Code: client redirect resource owner to auth server which in
turn redirect the resource owner back to the client with the Auth code
- Access Token: creds used to access protected resource
- Refresh Token: creds to obtain access tokens
- Redirects (generally 302, but up to server)
*** Client Registration
- not really part of the spec. It is up to us to find a way to add a list of
authorized clients, accepted redirection URIs, etc...
- Client Type
+ confidential: private server
+ public: executing itself on some device used by the resource owner
- Client Identifier
+ provided by the authorization server to the client
+ It is not secret
+ Unique
- Client password
can be used via Basic Auth, or in req params: ~client_id=...&client_secret=...~
Auth server MUST protect agains brute force attacks
#+BEGIN_SRC HTTP
POST /token HTTP/1.1
Host: server.example.com
Content-Type: application/x-www.form-urlencoded
grant_type=refresh_token&refresh=...&client_id=...&client_secret=...
#+END_SRC
OAuth2 doesn't support unregistered clients
*** Protocol Endpoints
Two auth server endpoints:
- Authorization endpoint: used by the client to obtain authorization from the
resource owner via redirection
- Token endpoint: used by the client to exchange an authorization grant for an
access token, typically with client auth.
Client endpoint:
- Redirection endpoint: used by the auth server to return responses containing
auth creds to the client via the resource owner user-agent
**** Authorization Endpoint
- MUST ignore unrecognized parameters
- MUST not accept repeated params
Client must inform the client of the =response_type=:
- must be set to ~code~ for requesting auth code
- must be set to ~token~ for implicit grant
**** Redirection Endpoint
After completing its interaction with the resource owner, the auth server
directs the resource owner's user-agent to the client.
The redirection URI MUST NOT include a fragment component.
Registration requirement: the auth server MUST require the public and
confidential clients to register their redirection endpoint.
For our usage: the end point must use TLS (https) and also there should be a
single URI per client.
If an auth fails due to missing, invalid or mismatching redirection URI.
The Auth Server should inform the resource owner without redirection.
**** Scope
An unordered list of space-delimited, case-sensitive strings.
The scopes are defined by the auth-server.
If requested scopes by the client are different by the ones granted by the auth
server. The auth server must return the granted scopes as a response parameter.
*** Obtaining Authorization
4 different Grants:
- authorization code
- implicit (no)
- resource owner creds (no)
- client creds (no user, direct client to IROH)
- an extension for more grant types.
**** Code Grant
Used to obtain both access token & refresh tokens.
Optimized for confidential clients.
1. client construct the request URI with the following params:
+ ~response_type~ REQUIRED. Must be ~code~.
+ ~client_id~
+ ~redirect_uri~ (OPTIONAL)
+ ~scope~ (OPTIONAL)
+ ~state~ (RECOMMENDED)
Example:
#+BEGIN_SRC
GET /authorize?response_type=code&client_id=s6BhdRkqt3&state=xyz
&redirect_uri=https%3A%2F%2Fclient%2Eexample%2Ecom%2Fcb HTTP/1.1
Host: server.example.co
#+END_SRC
2. Authorization Response. If auth is granted, then the auth server issues an
authorization code and delivers it to the client by adding the following
parameters to the query component of the redirection URI using the
"application/x-www-form-urlencoded" format.
+ ~code~ REQUIRED. auth code generated. Must expire shortly (10 min
recommended) and shouldn't be used more than once. If the code is used more
than once, should revoke all tokens previously issued based on that
authorization code.
+ ~state~ REQUIRED. Same value as the client provided.
3. Client makes a request to the token endpoint. Should contains the following params:
+ ~grant_type~ REQUIRED. must equal to ~authorization_code~.
+ ~code~ REQUIRED. The auth code received.
+ ~redirect_uri~ REQUIRED. Same as in the authorization request.
+ ~client_id~ REQUIRED. If the client is not authenticating with the auth server.
Example:
#+BEGIN_SRC
POST /token HTTP/1.1
Host: server.example.com
Authorization: Basic czZCaGRSa3F0MzpnWDFmQmF0M2JW
Content-Type: application/x-www-form-urlencoded
grant_type=authorization_code&code=SplxlOBeZQQYbYS6WxSbIA
&redirect_uri=https%3A%2F%2Fclient%2Eexample%2Ecom%2Fcb
#+END_SRC
4. Access Token Response.
If the access token is valid and authorized, the auth server issues an access
token and optional refresh token.
Example:
#+BEGIN_SRC
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
Cache-Control: no-store
Pragma: no-cache
{
"access_token":"2YotnFZFEjr1zCsicMWpAA",
"token_type":"example",
"expires_in":3600,
"refresh_token":"tGzv3JOkF0XG5Qx2TlKWIA",
"example_parameter":"example_value"
}
#+END_SRC
** OAuth2 Provider Epic
https://github.com/threatgrid/iroh/issues/1349
*** Functional Spec
We want 3rd party to be able to use our APIs (IROH + AMP Global Intel) by making
an OAuth2 Provider.
*** Tasks
See next sections for technical details.
=[#A]=, =[#B]=, =[#C]= are priority order to be able to test the OAuth system
before finishing all the tasks:
- =[#A]= Be able to provide keys to access our API
- =[#B]= Be able to migrate Visibility to OAuth2
- =[#C]= Support revocations, restrictions, etc...
Task without any set priority are lower priority.
*[[client-db][Client DB]]*:
- [ ] Write SQL store
- [ ] =[#C]= Write a service to manage clients
- [ ] =[#C]= Write routes restricted to IROH-Admins to manage OAuth2 clients
- [ ] =[#C]= POST =/clients= (the server sets the ~:id~ and returns the ~:password~
in clear text, in the DB the password should be encrypted).
- [ ] =[#C]= GET =/clients= (the ~:password~ field should be obfuscated)
- [ ] =[#C]= GET =/clients/:id= (the ~:password~ field should be obfuscated)
- [ ] =[#C]= PUT =/clients/:id= to revoke a client access (the ~:password~ field should be obfuscated)
- [ ] =[#C]= DELETE =/clients/:id= to revoke a client access (the ~:password~ field should be obfuscated)
*[[token-db][Authorization DB]]*:
- [ ] =[#B]= Write a service to manage "apps" for a resource owner
- [ ] =[#B]= Write routes to manage "apps" for a resource owner.
The =refresh-token= field should always be hidden or obfuscated.
- [ ] =[#B]= GET =/oauth/apps= returns the list of authorized apps
- [ ] =[#B]= GET =/oauth/apps/:app-id= return the authorized app informations
- [ ] =[#B]= DELETE =/oauth/apps/:app-id= remove the authorized app, further refresh
token queries should fail.
*[[user-db][User DB]]*:
- [ ] =[#C]= Write a Service to manage Users with the following methods:
- [ ] =[#C]= Create an user and generate its id. To be used for new login of still unkown user.
- [ ] =[#C]= search an user by internal id or 3rd party User id (typically AMP user id)
- [ ] =[#C]= revoke an user
- [ ] =[#C]= change scopes of an user
- [ ] =[#C]= Write routes restricted to IROH-Admins to manage Users
- [ ] =[#C]= GET =/users= returns a list of users
- [ ] =[#C]= GET =/users/:user-id= returns the specific user infos
- [ ] =[#C]= PUT =/users/:user-id= update an user, either revoke access or change scope
*[[routes][OAUTH2 Routes]]*:
- [ ] =/authorize= endpoint (should be behing the classic JWT protection)
- [ ] =[#A]= Support for authorization code grant
- [ ] =[#B]= Support for client creds grant
- [ ] Support implicit grant (close the current workflow with /login)
- [ ] =[#A]= =/token= endpoint
*[[middleware][Protected Resource Middleware]]*:
- [ ] =[#A]= Update the JWT middleware to only accept access-tokens
- [ ] =[#B]= Add a :scopes to compojure-api routes to filter access to JWT granting those scopes
- [ ] =[#B]= Update all routes to add a :scopes (cf. [[scopes][scopes]])
*[[frontend][Frontend]]*:
- [ ] =[#C]= Update the frontend to use the new OAuth2 routes.
- [ ] Create a dahsboard to help an user to revoke apps
- [ ] Create an IROH-Admin only dahsboard to manage clients (create, revoke, etc...)
- [ ] Create an IROH-Admin only dahsboard to manage users (revoke, etc...)
*[[doc][Documentation]]*:
- [ ] Documentation for making trusted daemons (authorization code grant)
- [ ] Documentation for frontend dev (implicit grant)
- [ ] Documentation for scripts (client creds grant)
*** Technical Spec
Write an OAuth2 provider.
Specificity of our OAuth2 provider.
- codes, access-tokens & refresh-tokens should be JWT.
Because JWT expires, and we can control how often we check their status
(revoked for example) server side.
Supported Grants (by priority order):
- ~authorization code~ grant
- ~implicit~ grant (for Visibility/iroh-ui) (similar to today =/login=)
- ~client creds~ grant
**** Client DB
<<client-db>>
We need to be able to add new OAuth2 "clients".
And we must be able to revoke a client.
So we need to add another store for clients.
Actually we shouldn't store those informations in ES.
We need to write an SQL backed CRUD store.
More specifically we'll use postgresql.
There should be another web-service exposing routes only for IROH admins.
*Question*: How should we keep client password?
*Proposal*: Present password in cleartext to the "creator" only once but save
the password using bcrypt.
A client MUST have a ~client-type~, a ~client-id~ and a ~client-password~.
Proposed internal representation:
#+BEGIN_SRC clojure
(s/defschema CRUDMetas
{:created-at s/DateTime
(s/optional-key :updated-at) s/DateTime
(s/optional-key :activated-at) s/DateTime
(s/optional-key :approved-at) s/DateTime
(s/optional-key :enabled-at) s/DateTime
(s/optional-key :blocked-at) s/DateTime})
(def Scope s/Str) ;; MUST NOT contain any space see the Scope section
(def URI s/Str) ;; shouldn't contains any fragment '#...', space or '..'
;; HIGHLY RECOMMENDED should be HTTPS
(defschema Client
(st/merge {:client-type (s/enum :confidential :public)
:grants #{(s/enum :auth-code :implicit :client-creds)}
:redirects #{URI}
:id s/Str
:password s/Str
:name s/Str
:scopes #{Scope}
:approved? s/Bool
:enabled? s/Bool}
CRUDMetas)
#+END_SRC
For example:
#+BEGIN_SRC clojure
(def threatgrid-client
{:client-type :confidential
:grants :auth-code
:redirects ["https://threatgrid.cisco.com/"]
:id "5fb04b9a-faad-11e7-8c3f-9a214cf093ae"
:password "70D3C005435E0F1C75A622C6C5AE92DB395A9E3140DE6980AC733AC9316AC0AE"
:name "ThreatGrid"
:scopes ["response" "enrich" "iroh-collect" "ui-settings" "ctia"]
:approved? true
:enabled? true
:created-at "2018-01-20T10:04:33"
:updated-at "2018-01-20T10:04:33"
:enabled-at "2018-01-20T10:04:33"
:activated-at "2018-01-20T10:04:33"
:approved-at "2018-01-20T10:04:33"})
#+END_SRC
**** User DB
<<user-db>>
We need to have an OAuth2 oriented Internal User DB.
For example to be able to change the scopes or simplly to revoke access to some
users.
Even if in the future we'll need a more complex User DB, for now we only need to
keep track of relation between an user-id for some 3rd party and an internal
user-id.
#+BEGIN_SRC clojure
(s/def IdPMapping
(st/merge
{:idp (s/enum :amp :threatgrid)
:user-id s/Str
:org-id s/Str}
{s/Any s/Any}))
(s/def UserClaims
{"cisco.com/iroh/user/id" s/Str
"cisco.com/iroh/user/name" s/Str
"cisco.com/iroh/org/id" s/Str
"cisco.com/iroh/org/name" s/Str
"cisco.com/iroh/roles" [s/Str]
;; see IROH-Auth JWT fields (remove all time-related ones)
})
(s/def User
(st/merge
{:enabled? s/Bool
:external-infos #{IdPMapping}}
PartialJWT
CRUDMetas))
#+END_SRC
**** Authorization DB / App DB
<<token-db>>
A refresh token are credentials representing the access granted by an user to
some client.
☞ Refresh tokens are typically long-lasting credentials used to request additional
access tokens, the refresh token is bound to the client to which it was issued.
(See https://tools.ietf.org/html/rfc6749#section-6)
So we should provide a mean for an user to revoke those accesses.
Remark the db should be populated via the =/authorize= endpoint.
There shouldn't be any way for an user to create or update an app, only delete.
Example:
#+BEGIN_SRC clojure
(s/defschema OAuthApp
{:id s/Str
:refresh-token s/Str
:client-id s/Str
(s/optional-key :client-name) s/Str
:authorized-date s/DateTime
:scopes #{Scope}})
#+END_SRC
**** OAUTH2 Routes
<<routes>>
We need to support the Code Grant so here is details about the workflow.
Note a lot of important details are in the RFC about how to handle those routes
securely.
Also, examples were modified so that the tokens are JWT.
***** Authorization Code Grant Worflow
The full details can be seen in the RFC: https://tools.ietf.org/html/rfc6749#section-4.1
Here some details were changed to adapt to our specifities (like using JWT).
The workflow is used to obtain both *access token* & *refresh tokens*.
Optimized for confidential clients.
1. client construct the request URI and redirect the
resource owner's user-agent to =/authorize= with the following params:
+ ~response_type~ REQUIRED. Must be ~code~.
+ ~client_id~
+ ~redirect_uri~ (OPTIONAL)
+ ~scope~ (OPTIONAL)
+ ~state~ (RECOMMENDED)
Example:
#+BEGIN_SRC
GET /authorize?response_type=code
&client_id=5fb04b9a-faad-11e7-8c3f-9a214cf093ae
&state=xyz
&redirect_uri=https%3A%2F%2Fclient%2Eexample%2Ecom%2Fcb HTTP/1.1
Host: server.example.co
#+END_SRC
2. Authorization Response. If auth is granted, then the auth server issues an
authorization code and delivers it to the client by adding the following
parameters to the query component of the redirection URI using the
"application/x-www-form-urlencoded" format.
+ ~code~ REQUIRED. auth code generated. Must expire shortly (10 min
recommended) and shouldn't be used more than once. If the code is used more
than once, should revoke all tokens previously issued based on that
authorization code.
+ ~state~ REQUIRED. Same value as the client provided.
3. Client makes a request to the token endpoint. Should contains the following params:
+ ~grant_type~ REQUIRED. must equal to ~authorization_code~.
+ ~code~ REQUIRED. The auth code received.
+ ~redirect_uri~ REQUIRED. Same as in the authorization request.
+ ~client_id~ REQUIRED. If the client is not authenticating with the auth server.
Example:
#+BEGIN_SRC
POST /token HTTP/1.1
Host: server.example.com
Authorization: Basic czZCaGRSa3F0MzpnWDFmQmF0M2JW
Content-Type: application/x-www-form-urlencoded
grant_type=authorization_code
&code=eyJh...SplxlOBeZQQYbYS6WxSbIA
&redirect_uri=https%3A%2F%2Fclient%2Eexample%2Ecom%2Fcb
#+END_SRC
4. Access Token Response.
If the access token is valid and authorized, the auth server issues an access
token and optional refresh token.
Example:
#+BEGIN_SRC
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
Cache-Control: no-store
Pragma: no-cache
{
"access_token":"eyJh...2YotnFZFEjr1zCsicMWpAA",
"token_type":"example",
"expires_in":3600,
"refresh_token":"eyJh..tGzv3JOkF0XG5Qx2TlKWIA",
"example_parameter":"example_value"
}
#+END_SRC
☞ *Refres Token*: for this PR, refreshing a token shouldn't create another
=refresh-token=. That way our client won't need to update their credentials. But
that is an option to keep in mind.
***** IROH Specificities
In IROH we don't have internal user login/password.
But we have a =/login= route that redirect with a =jwt= containing credentials.
So if we reach =/authorize= with such a valid =jwt= then we could continue by
using the user creds, if on the other hand we don't have such =jwt= the
=/authorize= should make a "double" redirection.
One to =AMP= (or =TG= in the future) login page and then the user-agent should
be redirected with the same URI but this time, a JWT should be provided in the
Headers if possible.
And then, if the =authorize= is validated, it will redirect to the client.
***** Tokens/Code format
One easy way to handle the lifetime of tokens and code is to use JWT.
During the code grant, the call to =/authorize= should redirect the user to some
page where the user interact and grant the access to the client. The output of
this call, the user is redirected via its user-agent with a short-living "code".
#+BEGIN_SRC clojure
(s/defschema IROHJWT ...) ;; see iroh-auth current JWT format
(s/defschema OAuthAuthorizeCode
(st/merge IROHJWT
{:kind (s/enum :grant)
:grant-type (s/enum :authorization-code)
:client-id s/Str}))
#+END_SRC
Such JWT MUST be REFUSED if used directly to access a protected resource.
Such =OAuthAuthorizedCode= should only be used to access the =/token= endpoint.
The =/token= endpoint will in turn returns two JWT.
The /access-token/ and the /refresh-token/.
- /access-tokens/ are the only token which grant access to protected resources.
- /refresh-tokens/ can be used to get /access-tokens/ without resource-owner interaction.
Proposed schemas:
#+BEGIN_SRC clojure
(s/defschema OAuthAccessToken
(st/merge IROHJWT
{:kind (s/enum :access-token)
:user-id s/Str
:client-id s/Str
:scopes #{Scope}))
(s/defschema OAuthRefreshToken
(st/merge IROHJWT
{:kind (s/enum :refresh-token)
:app-id s/Str
:user-id s/Str
:client-id s/Str
:scopes #{Scope}}))
#+END_SRC
So when asking for a refresh token we could check the statuses of the user,
the client and the refresh-token.
**** Protected Resource Middleware
<<middleware>>
The current JWT middleware should be modified to only accept =:access-token= JWT
for protected resources routes.
There should more specific JWT system to handle the =/authorize= and =/token= routes.
**** Frontend
<<frontend>>
***** Become OAuth2 compliant
Visibility/IROH-UI should also become a "client".
And thus some minor changes should be done.
Instead of going to =/login= Visibility should access =/authorize= with the
following parameters:
- ~response_type~ value MUST be ~token~
- ~client_id~ the client id for visibility
- ~redirect_uri~ (as it is today)
- ~scope~ (the list of scopes Visibility need to access)
- ~state~ (the inner state of the app, the parameter as another name today)
The response format will also be slightly changed.
The fragment of the URL will contain:
- ~access_token~ (previously was ~jwt~)
- ~token_type~ should be ~bearer~
- ~scope~ OPTIONAL if identical to the scope requested by the client; REQUIRED otherwise
- ~state~ the state provided in the request
It is up to the server to make the same workflow than with the previous =/login=
***** App Dahsboard
Make an app dashboard to help users manage their authorized apps.
It should be inspired by the features of the API.
Only ability to revoke/delete some app.
***** IROH-Admin only dahsboard to manage clients
Have a dashboard for IROH-Admin only to manage clients.
Typically block accesses to a client but also change the client properties.
***** Create an IROH-Admin only dahsboard to manage users (revoke, etc...)
Have a dashboard for IROH-Admin only to manage users and be able to block some.
**** Scopes
<<scopes>>
While we can easily update scopes in the future.
We should start with a sane convention.
A scope should match the following regex: ~\w+[\w\d_-]*(/\w+[\w\d_-]*)*(:\w+[\w\d_]*)?~
A nicer way to look at it is that it should have the following format:
- =<high-level-api>=
- =<high-level-api>(:<access-type>)?=
- =<high-level-api>(/subset)*(:<access-type>)?=
Mainly identifier with potential =:= followed by another identifier.
The main format is: =<api>= or =<api>:<access-restriction>=.
For example: ~enrich~, ~enrich:read~, ~enrich/observe~ or ~enrich/observe:read~.
In order to filter access, if a JWT contains the following scopes:
#+BEGIN_SRC clojure
["ui-settings","enrich:read","collect/inspect"]
#+END_SRC
The mean the JWT can access all the routes with the following scopes:
- ~ui-settings:read~
- ~ui-settings:write~
- ~enrich/observe:read~
- ~enrich/enrich:read~
- ~enrich/deliberate:read~
- ~enrich/settings:read~
- ~collect/inspect:read~
But shouldn't access the following scopes:
- ~auth/client-mgmt:write~
- ~enrich/enrich:write~
- ~enrich/settings:write~
- ~collect/config~
Here is the exhaustive list of our exposed web services and the proposed scope prefix:
| route | scope |
|-------------------+-------------|
| /iroh-ui-settings | ui-settings |
| /iroh-auth | auth |
| /iroh-auth-mgmt | auth/mgmt |
| /fake-iroh-auth | auth/fake |
| /iroh-enrich | enrich |
| /iroh-response | response |
| /iroh-inspect | inpect |
When the route change some inner state (like writing to some store/db, then we
should use the ~write~ access-restriction. Otherwise the access restriction
should be ~read~.
Depending on the routes we should also add subsets. Typically:
- ~auth/mgmt/jwt-revoke~
- ~auth/mgmt/saml-org-whitelist~
☞ *Scope Accesses*: the scopes granted to an access-token MUST always be a
subset of the intersection of the scopes of the client and the scopes of the
user.
**** Documentation
<<doc>>
We should write a clear documentation to help 3rd party developer use our API
via OAuth2.
** OAuth2 Epics (3rd pass)
*** SPA compatible OAuth2
Provide a way for any website to use OAuth2 and make call to IROH API. Typically
make it easy to create a generic casebook lib or even perhaps a web browser
plugin that could work on any website.
**** Issues
- [ ] Support Auth-code without client password
- [ ] Support CORS that use informations from the access token
- [ ] Documentation:
- [ ] Write a document/blog post for developers
- [ ] Make a demo website if possible using a 3rd party lib like hello.js
- [ ] Provide a lib (for example a module for hello.js)
*** User's made OAuth2 clients
Users should be able to create OAuth2 clients with some limitations regarding
response and passwordless client.
IROH Admin users should create OAuth2 clients without scope limitation and
should be able to create public clients (destined for SPA).
**** Issues
- [ ] Decide if we authorize any user to create a public OAuth2 client that are
needed for SPA clients (see https://tools.ietf.org/html/rfc6749#section-2.1)
- [ ] User can access a webpage to manage their OAuth app grants #iroh-ui
- [ ] Provide the ability for user to create OAuth2 clients
- [ ] Forbid any client to require any `response/*:write` scope
- [ ] Provide urls to make all response using HTLM pages generated by Visibility
*** Internal User Representation
Currently all the data about an user is provided via the generation of IROH Auth JWT.
This kind of limit our ability to manage user access.
For example, while we can currently revoke access to any user given its id it would be
difficult to limit an user scope.
Also without an internal user representation it is not possible to merge
different IdP for the same user.
In the future it might also be useful to have Visibility only user preferrences.
**** Issues
- [ ] Write a documentation about how we should manage internal representation.
Also how to deal with default org, preferred IdP, etc...
- [ ] Write IROH admin-only, routes and service to
- create an user and generate its id
- search an user by either internal id or other 3rd party user id (typically
AMP or TG user id)
- revoke access to an user
- change authorized scopes of an user
*** OAuth2 Client Credentials Grant
Support OAuth2 Client Credentials grant which is really useful for scripts.
Mainly the workflow should be:
1. The user create a new confidential client with only client credential grant.
2. The user is returned with a client-id / client-password
3. The script can use those client creds to retrieve access only for that user.
- [ ] =[#C]= Write a Service to manage Users with the following methods:
- [ ] =[#C]= Create an user and generate its id. To be used for new login of still unkown user.
- [ ] =[#C]= search an user by internal id or 3rd party User id (typically AMP user id)
- [ ] =[#C]= revoke an user
- [ ] =[#C]= change scopes of an user
- [ ] =[#C]= Write routes restricted to IROH-Admins to manage Users
- [ ] =[#C]= GET =/users= returns a list of users
- [ ] =[#C]= GET =/users/:user-id= returns the specific user infos
- [ ] =[#C]= PUT =/users/:user-id= update an user, either revoke access or change scope
**** Issues
- [ ] Support Client Credentials Grant (for confidential client only, see
https://tools.ietf.org/html/rfc6749#section-4.4)
***** Client DB
- [ ] =[#C]= Write a service to manage clients
- [ ] =[#C]= Write routes to manage OAuth2 clients
- [ ] =[#C]= POST =/clients= (the server sets the ~:id~ and returns the ~:password~
in clear text, in the DB the password should be encrypted).
- [ ] =[#C]= GET =/clients= (the ~:password~ field should be obfuscated)
- [ ] =[#C]= GET =/clients/:id= (the ~:password~ field should be obfuscated)
- [ ] =[#C]= PUT =/clients/:id= to revoke a client access (the ~:password~ field should be obfuscated)
- [ ] =[#C]= DELETE =/clients/:id= to revoke a client access (the ~:password~ field should be obfuscated)
***** User DB
- [ ] =[#C]= Write a Service to manage Users with the following methods:
- [ ] =[#C]= Create an user and generate its id. To be used for new login of still unkown user.
- [ ] =[#C]= search an user by internal id or 3rd party User id (typically AMP user id)
- [ ] =[#C]= revoke an user
- [ ] =[#C]= change scopes of an user
- [ ] =[#C]= Write routes restricted to IROH-Admins to manage Users
- [ ] =[#C]= GET =/users= returns a list of users
- [ ] =[#C]= GET =/users/:user-id= returns the specific user infos
- [ ] =[#C]= PUT =/users/:user-id= update an user, either revoke access or change scope
***** Frontend
- [ ] Create an admin UI for any user to manage OAuth clients
*** IROH Admin Dashboard
It would be very effective to have a dashboard instead of just using the
swagger-ui. The dashboard would make it easy to centralize the access of all
informations as well as administration tasks.
Typically:
- be able to see some IROH logs filtered by some keywords, and be able
to revoke an user or an OAuth client. Also be able to change the scopes of some
clients.
- Provide the ability to force all user to be prompted to grant access again to
some client, etc...
*** OAuth2 Enhancements
Many OAuth2 feature should be enabled in order to provide the best security.
- [ ] Automatically revoke refresh token access if the code was used more than once.
Archived entries from file /Users/yaesposi/.deft/Cisco.org.gpg
* [#C] Scopes Dictionary
:PROPERTIES:
:ARCHIVE_TIME: 2019-01-09 Wed 10:53
:ARCHIVE_FILE: ~/.deft/Cisco.org.gpg
:ARCHIVE_OLPATH: Epics
:ARCHIVE_CATEGORY: Cisco.org
:END:
Regarding access right of API consumer this is enforced via scopes.
There are low-level scopes. Mostly one for each route. To help represent
coherent groups the scopes are organized as a tree whose leaves are tagged
either by read or write.
For example if an user has the scopes ~["foo/bar:read" "baz"]~
He'll could access all routes with those scopes:
- ~foo/bar:read~
- ~foo/bar/baz:read~
- ~foo/bar/baz/quux:read~
- ~baz:write~
- ~baz/x/y:write~
But he won't be granted the access to:
- ~bad:read~
- ~foo/bar:write~
- ~foo/bar/baz:write~
The problem with this organisation is that it's somehow too technical. This is
why we need to introduce high-level abstracts scopes to give meaningful names to
precise subsets of route access.
To that end we should provide a dictionnary (with schema
~{s/Str {:scopes #{s/Str}} :name s/Str :description s/Str}~):
#+BEGIN_SRC clojure
{"importer" {:scopes #{"collect" "ctia:read"}
:name "Importer"
:description "This client would like to be able to import events into
your private intel store,
and needs some limited read permissions as well"}
...
}
#+END_SRC
Technically it would simply mean that if an user has the scope "importer" we'll look into that
dictionnary to provide him the technical scopes.
Furthermore we'll be able to look into this dictionary to provide a meaninful name and description
to the user when he's asked to grant some access to some 3rd party app (OAuth client).
Archived entries from file /Users/yaesposi/.deft/Cisco.org.gpg
* DONE Document for Raghavaiah
CLOSED: [2018-11-28 Wed 18:01]
:PROPERTIES:
:ARCHIVE_TIME: 2019-01-09 Wed 10:54
:ARCHIVE_FILE: ~/.deft/Cisco.org.gpg
:ARCHIVE_CATEGORY: Cisco.org
:ARCHIVE_TODO: DONE
:END:
** Frank's proposal; Auth config untangling
IROH Auth and SSE staging config changes to correctly link environments and improve naming
*** INT
**** idb-amp-staging / selector: amp
This is actually pointing to QA1 on the AMP side. We should name it as such.
- CTR
- Change identifier to `idb-amp-qa1`
- Change name to `AMP QA1`
- Change selector to `amp-QA1`
- SSE IDB
- Change selector to `amp-qa1`
**** idb-tg-staging / selector: tg
This is TG INT, just need a rename
- CTR
- Change identifier to `idb-tg-int`
- SSE IDB
- CHange selector to `tg-int`
*** TEST
**** idb-amp-staging / selector: amp
This needs to go to AMPs SDC environment instead. We'll need a new auth provider setup on the SSE side for AMP SDC
- CTR
- Change identifier to `idb-amp-sdc`
- Change name to `AMP SDC`
- Change selector to `amp-sdc`
- SSE IDB
- New auth provider with URL: https://auth-ext.qa1.immunet.com/
- Selector `amp-sdc`
**** idb-tg-staging / selector: tg
This will need to point at the TG TEST environment, not INT. It'll also need a new auth provider be created in the IDB
- CTR
- Change identifier to `idb-tg-test`
- Change name to `Threat Grid TEST`
- Change selector to `tg-test`
- SSE IDB
- New auth provider with URL: https://test.threatgrid.com/ (Frank to create new oauth2 client if needed)
- Selector`tg-test`
** Proposal
To help people manage all the IDB clients and connection we would like to follow a stricter naming convention.
There are many different named entities:
- IROH IdP id
- IDB selector
- IROH-UI button label
The IROH IdP id is used in the =redirect_uri= that should be configured in the
IDB client.
Also there are two different IDB environment. IDB prod and IDB stage.
Each client should also point to some IdP environment, typically, AMP NAM Prod,
or AMP QA1, or AMP SDC.
The namin convention should make it clear what IDB instance is used as well as
with IdP it targets, in particular we should make each region explicit.
Note all point prefixed by =IDB= are the information needed by the SSE/IDB team.
*** INT
**** AMP
Note: I think it should point to AMP prod as before to prevent the use of the VPN
to login to INT
- IROH IdP id: idb-stage-amp-nam
- IDB env: Stage
- IDB selector: idb-stage-amp-nam
- IDB redirect uris:
- https://visibility.int.iroh.site/iroh/iroh-auth/login/idb-stage-amp-nam/answer
- https://localhost:5443/callback
- IDB IdP Target: https://auth.amp.cisco.com/auth/session/new
- IROH-UI button label: Cisco Security
**** TG
- IROH IdP id: idb-stage-tg-int
- IDB env: Stage
- IDB selector: idb-stage-tg-int
- IDB redirect uris:
- https://visibility.int.iroh.site/iroh/iroh-auth/login/idb-stage-tg-int/answer
- https://localhost:5443/callback
- IDB IdP Target: https://int.threatgrid.com/oauth2/authorize
- IROH-UI button label: Cisco Security
*** PROD NAM
**** AMP
- IROH IdP id: idb-amp-nam
- IDB env: Prod
- IDB selector: idb-amp-nam
- IDB redirect uris:
- https://visibility.amp.cisco.com/iroh/iroh-auth/login/idb-amp-nam/answer
- IDB IdP Target: https://auth.amp.cisco.com/auth/session/new
- IROH-UI button label: Cisco Security
**** TG
- IROH IdP id: idb-tg-nam
- IDB env: Prod
- IDB selector: idb-tg-nam
- IDB redirect uris:
- https://visibility.amp.cisco.com/iroh/iroh-auth/login/idb-tg-nam/answer
- https://visibility.apjc.amp.cisco.com/iroh/iroh-auth/login/idb-tg-nam/answer
- https://visibility.test.iroh.site/iroh/iroh-auth/login/idb-tg-nam/answer
- IDB IdP Target: https://panacea.threatgrid.com
- IROH-UI button label: Threat Grid
*** PROD EU
**** AMP
- IROH IdP id: idb-amp-eu
- IDB env: Prod
- IDB selector: idb-amp-eu
- IDB redirect uris:
- https://visibility.eu.amp.cisco.com/iroh/iroh-auth/login/idb-amp-eu/answer
- https://visibility.test.iroh.site/iroh/iroh-auth/login/idb-amp-eu/answer
- IDB IdP Target: https://auth.eu.amp.cisco.com/auth/session/new
- IROH-UI button label: Cisco Security
**** TG
- IROH IdP id: idb-tg-eu
- IDB env: Prod
- IDB selector: idb-tg-eu
- IDB redirect uris:
- https://visibility.eu.amp.cisco.com/iroh/iroh-auth/login/idb-tg-eu/answer
- https://visibility.test.iroh.site/iroh/iroh-auth/login/idb-tg-eu/answer
- IDB IdP Target: https://panacea.threatgrid.eu
- IROH-UI button label: Threat Grid
*** PROD APJC
**** AMP
- IROH IdP id: idb-amp-apjc
- IDB env: Prod
- IDB selector: idb-amp-apjc
- IDB redirect uris:
- https://visibility.apjc.amp.cisco.com/iroh/iroh-auth/login/idb-amp-eu/answer
- https://visibility.test.iroh.site/iroh/iroh-auth/login/idb-amp-apjc/answer
- IDB IdP Target: https://auth.apjc.amp.cisco.com/auth/session/new
- IROH-UI button label: Cisco Security
**** TG
This one is the same as the TG for prod. TG doesn't have an APJC endpoint for login.
- IROH IdP id: idb-tg-nam
- IDB env: Prod
- IDB selector: idb-tg-nam
- IDB redirect uris:
- https://visibility.amp.cisco.com/iroh/iroh-auth/login/idb-tg-nam/answer
- https://visibility.apjc.amp.cisco.com/iroh/iroh-auth/login/idb-tg-nam/answer
- https://visibility.test.iroh.site/iroh/iroh-auth/login/idb-tg-nam/answer
- IDB IdP Target: https://panacea.threatgrid.com
- IROH-UI button label: Threat Grid
*** TEST
For test, we should want to be able to login through many different envs to
check none is broken or a future change won't break our login.
So the "classical integration with test envs".
**** TEST IdPs
***** AMP SDC
- IROH IdP id: idb-stage-amp-sdc
- IDB env: Stage
- IDB selector: idb-stage-amp-sdc
- IDB redirect uris:
- https://visibility.test.iroh.site/iroh/iroh-auth/login/idb-stage-amp-sdc/answer
- https://localhost:5443/callback
- IDB IdP Target: https://auth-ext.qa1.immunet.com/
- IROH-UI button label: AMP SDC
***** TG Test
- IROH IdP id: idb-stage-tg-test
- IDB env: Stage
- IDB selector: idb-stage-tg-test
- IDB redirect uris:
- https://visibility.int.iroh.site/iroh/iroh-auth/login/idb-stage-tg-test/answer
- https://localhost:5443/callback
- IDB IdP Target: https://test.threatgrid.com/oauth2/authorize
- IROH-UI button label: Cisco Security
**** Prod Logins
Instead of duplicate the infos, we should copy all buttons from the 3 prod regions.
Note it will only create 5 buttons instead of 6 because there is no TG APJC login page.
* Daily Standup Meeting
:PROPERTIES:
:ARCHIVE_TIME: 2019-03-15 Fri 09:49
:ARCHIVE_FILE: ~/.deft/Cisco.org.gpg
:ARCHIVE_OLPATH: Meetings
:ARCHIVE_CATEGORY: Cisco.org
:ARCHIVE_ITAGS: noexport_1
:END:
** <2019-02-27 Wed>
*** release 1.19
- There are a few blockers. About redirections. 52 PR in IROH Services. 13 PR for UI.
- #1044, login account, wrong link to AMP instead of TG if you login via TG.
We should be able to cut the release, and fix those during the release.
*** individual updates
**** Yann
Open Entitlement in progress.
**** Paula
Working on Asami. Tomorrow working in SF.
**** Rekha
Incident detail page.
**** Mike
Working with Nola on session.
Help Rekha with some bugs.
JWT session work today and help Rekha.
**** Matt
SMA module. Extract hash file and filename from message details from SMA.
**** Jesse
Incident related things. Working on speed issues.
**** Guillaume Ereteo
CTIA Investigate. New route in CTIA to use POST.
**** Anglea
Cut an ATS integration release.
Added a health call, writting up how to configure the module.
**** Alex
Merge Talos IP, a PR related to a cryptominer malware.
**** Mary
Working on help topics for the Umbrella module.
Trying to create the wireframe for help topics.
Going back to TG tomorrow.
*** UI/UX
Graph should be able for review on Friday.
*** Misc
Jesse:
- UI Staging broken, should be fixed
** <2019-02-25 Mon>
** <2019-01-18 Fri>
*** Individual update
**** Yann
- merged user lookup API
- long talk to make that better for the front, unfortunately difficult to
achieve
** <2019-01-23 Wed>
rel 1.16
ops report
design updates
individual updates
*** ops
Posted in IROH Ops channel
Everything seems ok
Be able to release? no config change, so should be ok
*** rel 1.16
- Blocker is still open iroh-ui#1030
- Blocker about doc
- Everything else look fine
*** Individual update
**** GE
Investigating a problem with Yann (test fail locally but not in CI)
**** Angela
working on ATS duplicate module fixes
we have some merging we have to do.
**** Paula
Working on with Jesse on a lib to be included in iroh UI.
Helping Mike with onboarding.
**** Jesse
follow up. #1030 I should look into it. Then back to incident manager
**** Matt
lot of rate limiting API, starting to introduce caching
**** Mike
I am trying to make the application run in my local machine
**** Nola
CVO is working.
Some PR this morning, refactor a bit for the Incident.
**** Yann
PR about split admin/non-admin fichinshed.
Should first fix the master test issue. Travis are ok, but local test fail.
*** UX Design update
Sam: will discuss with Craig to see if everythin on track.
Brian, New investigation page content? nothing special to comment on.
Open issues on other boards is for making sure other things are done.
*** Design update
** <2019-01-11 Fri>
*** Ops Weather Report
- NGFW Incident Manager in end of the month. Write speed limit in our
cluster. We should keep track of health.
- Lot of rate-limiting on Umbrella queries, we pull a list of domain from
the platform API. That is in the platform API.
*** Individual Reports
**** Alex
All of my dev env setup on the new laptop.
**** Angela
**** GE
- did my first PR, I have comments from Guillaume
**** Jesse
- Incident tickets (Incident Manager UI)
**** Matt
- dynamic resolution for SSE Proxy URL
**** Nola
...
**** Paula
react column resizing, sorting
**** Yann
- merged a PR yesterday for adding better logs of SSE HTTP calls
- waiting a review for a PR that fix the whoami with transit+json (pb with
dates)
- I think there are some work to be done to prepare for the split between
admin/non-admin roles in IROH. That will certainly help or be related to
the Open Enrolment. Like having unshared user's modules
*** Design
**** SAM
- Support Incident Manager
- Engaging users from Snehal
**** Brian
- Thanks to Guillaume to discuss with me yesterday
** <2019-01-09 Wed>
secretary: GE
*** Ops
- Looks good
- conf change by Matt
*** Reports
**** Angela
Introduction: setup dev env
**** Brian/Sam design update
I'm out of the CTR world.
Incident manager, detail fields.
**** G2
Work on CTIA investigate route
**** Lance
Support Incident manager
**** Matthieu
Dynamic resolution of URL for the API Proxy based on the SSE team recommendation
Working with Naasief for a lot of SMA issues
**** Nola
Miss CSS
**** Paula
Helper Jesse clear away the principal stuff with UI. Layout with incident happening.
Fixing some bugs with React (ticket number 1045), table column resizing.
**** Sam
Nothing to add
**** Yann
Working on issues.
**** Neel (QA)
Test cases for casebook
** <2019-01-08 Tue>
*** OPS
- quiet week
- details in the channel
*** Release Status
*** Angela
- Austin Office
- Face in Austin
** <2019-01-04 Fri>
- ops: quiet
Individual updates:
- Jesse: interaction diagram for incident manager
- Nola: environment setups (working on #1030)
- Paula: regression in cljs, implemented incident service in ticket #37
- Houman: pretty quiet for us, working on automation test
- Guillaume: triaging issues backlog
UX update:
- Samuel: Interview next week, need to sync up with Brian
Doc update:
- help topic to include un 1.16 for SMA, lot of help need to be done, lot of
draft must be converted to markdown, and working on the content of the module
details. Replace "configure module" by "add module". Also casebook help should
change a bit to handle application.
Make a specific page by user provenance on the homepage. Add first, then configure.
** <2018-12-07 Fri>
*** Topics
Paula secretary
- rel 1.15
- ops
- individual updates
- demo with Dean, Al, etc...
*** Ops
Nothing
*** Rel
- Pb with SMA, discussion with SSE team
- new branch should be done for the Frontend
** <2018-11-28 Wed>
*** release 1.14
- blocker yesterday, should be a fix
- something strange with the OAuth2 workflow
Alex: we've got a lot of discovered blocker 24h before launch.
wonder if we should change something
*** Ops report
all good
*** Individual Report
**** Nola
- Working with Mary
- Need to make a fix
- Jesse to clean the code
**** Jesse
- working toward support the incident
- SMA integration mostly there
Matt: all the code should be done, I can help
**** Matt
- Fix a bug on CTIA
- Tested crosslaunch, doesn't workf the 1st time.
- working on adding pagination
**** Yann
- SSE Tenant to JWT proxy
- IDB conf
- fixes
**** Alex
cssv2, to be able to push lot of new entities in ctia
*** UX Design Report
**** Samuel
- will help with documentation SMA
** <2018-11-26 Mon>
Craig: let's keep it really short.
** <2019-01-11 Fri>
** <2018-11-19 Mon>
*** Ops
- Some 500, French were reported about it
*** UI
Craig: metrics bars and other metrics remarks
*** Individual reports
**** Nola
- test failed for the login page due to scrolling.
- Craig: 5 buttons are more than we really ever deploy.
**** Paula
- Ticket #1000, working with Jesse this afternoon to track that down
**** Jesse
- found and fixed the Target bug
**** Alex
- wrapping up relationship updates
- working on turning some Umbrella data to CTIM bundles
**** Matt
- start to test SSE modules
**** Yann
- working to add the IDB from prod
**** Craig
- offer to someone to joins Jesse's team
- more interview for UI
- Freeze on wednesday: focus on the SSE integration
*** JWT lifetime
- Craig: Ask bump to 24h for id_token.
** <2018-10-31 Wed>
*** Individual Reports
**** Yann
- Long call with SSE team Monday
- working toward integrating the IDB
- two modifications, one approved
- for now every flags are GREEN
- Here to help QA regarding clients management
** <2018-10-22 Mon>
*** Individual Updates
Worked with Matt to write down the complete workflow we need to use for the IDB
& SSE integration.
** <2018-10-10 Wed>
- PR about ODNS Error handling in review
- PR about SAML Issuers to unblock AMP IdP team
- Client doc PR still open but accepted
- SSE Integration, thinking about how we could not use the IDB as our main login access.
- Will get the stats for Eduardo (Thanks Daniel)
** <2018-10-01 Mon>
- individual reports
- Fixed a bug from Houman (waiting for review)
- Enhanced the CSV format to be closer to the one requested by Craig
- Will work on OpenDNS error handling next
** <2018-09-19 Wed>
*** Ops weather reports
John: casebook 500 detected
Guillaume: certainly will be a fix during next release.
*** Release report
Houman:
- cherry-picked few PRs
- waiting a fix on the relation graph
- no more blocker
*** Doc report
Mary:
- Working on the 1.9 release notes
- Perhaps I would create a summary.
- talking with Sam with adding online icon to the top of the casebook.
- Created an issue for creating those user accounts
Working on a doc plan, where could I write it?
Jesse: put that in the wiki of iroh-ui repo.
*** Personal report
**** Yann
- admin routes: not accessible yet on INT, need Daniel to work on it
- fixed some scopes on some routes that didn't had any (app-grant)
Next: extend clients creation abilities
**** Jesse
- fixing the bug. Hopefully fix it this morning.
**** Lance
- wireframe for CTR Enterprise integration
**** Matt
- finish PR about rate limiting
**** Nola
- Added a new tab on the size for casebook, test failed on selenium.
- Keep working on that
- Modified the timeline to make it more simpler
**** Paula
- Going through bugs, replays the queues issue
*** UX Update
Sam: doing some review yesterday By the end of this week, everything will be
complete and we'll move to next issue.
*** PTO
- Jesse: next Monday
- Paula: next week will go to a conference
- Mary: long weekend on October
** <2018-09-18 Tue>
- ops report: green
- release status:
- IROH board to fix
- fix for #750 and Jesse on it
-
** <2018-09-11 Tue> Secretary: Nola
*** Topics
- Ops report
- PR merged
*** PR merge
- tomorrow cut release branch by the end of the day
*** PTO
Craig: a week out of october 1st
Mary: long weekend in october
Paula: last week in september going to Strangeloop
Lance: las week in september
** <2018-09-10 Mon>
*** Yann
- Reviewed rate limiting Matt's PR
- Will work on next steps regarding browser plugin support for OAuth2. More
precisely ability to change the lifetime of JWT for access-token,
refresh-tokens, etc..
- Also just introduced benchmark tool for tk-store and checking if schema
validation doesn't cost too much
*** UX Design
*** IdP Issue
Frank dynamic, so we can put.
** <2018-08-14 Tue>
- Sound
- OPS
** <2018-08-17 Fri>
*** Topics
- Ops report
- Individual reports
- UI/UX report
- PTO
*** Ops report
Everything went completely drama free.
Alex: we need to start monitoring closer to the app, not only system monitoring.
Maybe we can have some conversion how to achieve that.
Like cert errors because a module isn't configured, etc...
action: meet next week.
*** Individual Reports
**** Chris
- in progress: table unification Issue #666 in the response repo
- fixing bugs this week
**** Jesse
- working with Paula and graph changes
**** Nola
- working on #638 on the table, added the headers and stuff.
**** Paula
- doing some merging necessary to fix a bug.
- want to see all the graph work in master.
**** Yann
- Fix an OAuth client CRUD bug due to lack of coercion with Postgres backend.
- Another PR to generalise how we handle coercion and data validation with our stores.
- Fix a log messages
**** Alex
- Necessary testing for Nation DB importer
- Some changes need to be made to CTIM & CTIA.
*** UX Design Update
- wireframe reviewed & will check with.
Craig: I'll check with you to review the wireframe.
*** PTO
Yann: two weeks
Chris: certainly take about 2 days next week.
** <2018-08-13 Mon> Secretary Paula
*** Individual report
**** Yann
***** Enhanced Client Creation
Handle Admin vs Non admin creation also better security
Pushed another commit on top of the Client Creation PR
Not major, still not totally minor.
Mainly now, admin can create any kind of
clients, with response scope and empty password.
The empty password creation will make me possible to work next on describing how
to use OAuth2 for frontend.
I'll just miss the CORS handling from JWT work.
** <2018-08-08 Wed> Secretary John
*** Individual Report
**** Yann
***** Done
- Reviewed GB PR
***** Doing
- Write Developper Doc
***** Blocked
- merge PR waiting for Daniel return
** <2018-08-07 Tue>
- release: ok
- ops: ok
*** Brian
Question selenium tests & iroh-test, all major browsers easy to write in clojure!
** <2018-08-06 Mon>
*** Bart resigned (by mail)
Craig: Bart resigned Friday by mail. I wouldn't encourage to do that in the
future. Please call me first.
*** Individual report
**** Yann
- Cleaning up my PR, waiting on Ops
- added the missing expires_in in the OAuth2 /token response
minor but really all devs aked for it
- NEXT: certainly work on the Entitlement issue
OR missing admin routes (all users, all orgs, all clients, etc...)
**** Nola
- Friday, talked and watched video.
- Today deep dive with Alex on the system, added some comments on tickets
**** Samuel
- Friday, work with Bart & Chris about their PR
**** Alex
- More orientation, on vuln DB & presentation to Dean
** <2018-08-03 Fri>
*** Topic
- Individual report
- ops report
- PTO in Canada
*** Individual Update
**** Paula
Bug tracked down and fixed.
**** Chris
OAuth is ready to be reviewed (PR open)
**** Jesse
COA demo, almost accomplished particularly since Paula fixed the bug.
**** Matt
Reviewed Yann's PR.
**** Yann
***** Client Creds PR accepted
will make some feedback fixes and I'll wait for Daniel to be able to merge it.
**** Nola
Met with Bart, debugging with clojurescript. Date formatting issue. No test function for date function.
Jesse: sound familiar, can help
**** Samuel
Documentation
**** Bart
opened PR
*** Ops Report
- 1.6 out Nothing
- 1.7 next week
*** Release Status
Everything look good.
Monday Holliday in Canada, having the thumbs up/down next week on tuesday.
** <2018-08-01 Wed>
*** Topics
- Blocking Bug #622 in IROH-UI
- ETA on fix
- Can we cherry pick the commit into 1.6?
- Operations Deploy Status
- Are we ready to deploy?
- Report on SSH access to VMs
- Individual Reports
- What's underway
- What's wrapped up
- PTO
- Update from Craig
*** Update from Craig
- Back from Michigan yesterday.
- Defining what we're commiting to deliver.
- Already started to work on or naturally on our roadmap
- User Provisionning Entitlement
- Device manager support
- Incident manager
- Casebook browser plugin
- Suggested COA (major bit of work for the UI team)
- Customer Success Support
- Open APIs:
- Client Creds
- rate limiting and relay modules
- NGFW Incident Flow
- SMA/ESA Integration
- Umbrella Enrichment
- Threatgrid Enrichment
- basic informations and start from there.
Production by 12/22/2018
Filtering node aggregation to come in the UI, but we don't commit to this.
Bug fixes.
*** Individual Reports
**** Bart
Pair programming with Nola to fix bugs
**** Chris
Writing tests for OAuth clients
**** Guillaume
Working on Epics & TLP stuff. Adding tests
**** Paula & Jesse
Pairing with Jesse, getting the full integration to demonstrate the graph update
with rules. Doing simple PoC at the moment.
**** Matt
Write test for setting enrichment for Umbrella Reporting API
PR in review.
**** Nola
Working with Bart to fix bugs and working with clojurescript
**** Sam
back from UX summit and I document some things and connecting with Bart & Chris.
**** Yann
Waiting to merge PR, working on Client OAuth Grant.
**** Alex
Weakness importer is done and ready for review. Starting work on NVD importer.
*** Blocking Bug #622
Jesse made the fix and approved by Bart yesterday.
Alex would like the fix to be cherry-picked and merge by Houman and deployed in prod.
(the relation graph misbehave).
Craig: we want to be able to make hotfix.
Alex created a new issue to add a regression test for that issue. Houman is the
day to cut the 1.7. Houman: I can start the release today. All PR appear to have
QA informations or be straightforward.
*** Ops
John: We'll try to deploy to APJC later.
*** PTO
Guillaume: tomorrow and Friday.
** <2018-07-25 Wed> Secretary Matt
*** Individual Reports
**** Yann
***** Done
***** Doing
****** User / Org DB.
Still working on User and Org DB PR.
Lot of tests and I'll certainly ask questions to Craig regarding some choices.
***** Blocked / Waiting
***** Next
** <2018-07-24 Tue>
Topics:
- release
- ops report
- release notes
*** Release
Houman:
- tickets are almost done, I don't see any major issue or blocker.
- Manual tests on IE and Firefox.
*** Operation Report
Nothing to report.
This afternoon 3pm Mountain Time, we'll meet to check thumbs up/thumbs down for deploying the release.
*** Release Notes
Alex: We need a better way to make release note.
- ideas: tags, git logs
- who should be in charge of writing them?
We'll wait for Craig to be present to discuss that again.
** <2018-07-23 Mon> Secretary Alex
*** Individual Reports
**** Yann
***** Done
***** Doing
****** User / Org DB.
Work on User and Org DB PR. The DBs should be filled during Authentication
activities. Should provide Ability to block user/orgs easily.
***** Blocked / Waiting
***** Next
** <2018-07-17 Tue>
*** Topics
- release update
- Operations weather report / pre-Daniel vacation send-off
- PTO
- Data Staging server on INT
I didn't though I would take days in July.
But I've got family coming by surprise, so I'll take Thursday and Friday.
And perhaps, I won't be there tomorrow afternoon.
** <2018-07-16 Mon>
*** Topics
- cutting a release on Wednesday
*** Individual Reports
**** Yann
***** Done
- Helped Raghavaiah Avula from SSE Identity Team with Matt. Provided him the
information about how to create an OAuth2 client in TG.
- Merged the AMP Metadata URL. Stricter than before, now all SP-URI should be
checked. Fixed that with Chris.
- Created an Issue in Tenzin to deploy that metadata URL. As we don't have
strict delivery date, Daniel proposed to make that change for 1.7.
- Chris detected an issue with the SAML PR on INT. I fixed the conf on INT for
frontend team to continue be able to login using different url than
visbility.int.iroh.site but with ui-staging and localhost. I changed in the
K/V config of consul on INT.
***** Doing
- Must find a way to provide a list of SP URIs in the SAML Metadata URL
***** Blocked
***** Waiting
- OAuth Clients Grant, next step about filling User and Org DBs
**** Paula
- Cleaning up during the week-end.
- I'm out this week on wednesday and Thursday.
**** Matt
- Still working on SMA integration
- Starting to request them.
- Still what I'm doing.
- Lot of communication with SSE team, to see how to configure connectors.
**** Alex
- Working on XHTML->Markdown
** <2018-07-13 Fri>
- release go?
* Houman: good
* Ops: redeploy IROH
* Overhead by region?
- tenzin issue for pbs.
- OPs whishlist
- done: SAML metadata URL + Signing Request (PR open)
** <2018-07-11 Wed> Secretary: Bart
- individual report
- release deployment day
- conversation in the IROH channel
- soliciting topic
- impersonate API
- integrating with AMP test env
- update from John
*** Individual update
**** Alex
**** Jesse
Integrate fixes for Houman
**** Yann
Done Tasks:
- fixed a minor bug on Impersonate API
- helped Robert Harris yesterday for the auth of the Chrome Casebook plugin.
Apparenly the auth is good now.
- Kibana Dashboards with Matt
Actively Doing Tasks:
- Main topic, almost blocked for 3/4 days, find the right way to add the
metadata URL and sign the Authnrequest.
- incremental blockers task, looks to be working for some time, but fails at
first, second attempt, search another solution
- lack of documentation, lack of examples, lack of working clojure lib, lack
of working java examples, etc...
- not blocked anymore and should be clean
- Waiting task: Worked to integrate the ID Broker
- Interrupted task: Client Creds and adding Internal User DB & Org DB
**** Sam
Working on, diagram for a process for the UX flow, the vision to UX to dev.
*** Release Deployment Day
- Houman: found a blocker this morning, should be fixed by Jesse
*** conversation in the IROH channel
- We're not using hipchat to the same degree we did.
- action to avoid a few
*** soliciting topic
- more traction in soliciting topics
- increase the signal
- involve 30% of the group a topic, its fine to bring it up
- better context about what the rest of the team is doing
*** impersonate API
Frank: does it go to production.
VPN access only.
*** integrating with AMP test env
Configure Test and Prod as IdP
*** update from John
** <2018-07-09 Mon>
- Release (deploy week)
- PTO
*** Release
Houman: Few more tickets to finish
*** PTO
*** ...
Dean will have a Meeting this afternoon.
** <2018-07-06 Fri> Secretary Jesse
- release
- PTO
*** Release
no blocker, working on IROH Services issues
*** Discovery
Can use current authorization code flow.
Chrome Plugin to use Vis everywhere
*** Bart Demo
*** Upcoming PTO
Chris Wednesday, Thursday, Friday.
** <2018-07-04 Wed>
Topics:
- release will be cut today
- contains
** <2018-07-03 Tue>
- Manpreet meeting with FR
- Cutting release tomorrow
- Introduction of new Ops member: John Jardine
- chit chat, editor
- presentation tour
** <2018-06-29 Fri>
Topics:
- iroh-ui PRs
- PTO
Cut new release on Wednesday next week.
Probably unschedule that.
*** IROH UI
No blocking PR, quality is already high.
Focus: on bug fixes and user facing changing
Craig is relax.
*** PTO
** <2018-06-22 Fri> Secretary Yann
Topics:
- Release status
- Ops report
- TTP headnodes
- PTO updates
*** Release Status
- Houman: making good progress.
*** Ops Report
- Daniel: Checking logs daily, so far everthing going well.
- Alex: We should find a way to create reports about error logs. To be looked by
the team to have better insight about how users use our app. Kibana has some
limitations but we can get a lot of interresting informations.
*** TTP Headnodes
- Alex: I propose to create another CTIA instance to push new data into it so
its easier to delete indexes.
- Matt you can use private CTIA.
- Alex: easier to separate indexes.
*** PTO update
- Alex: Taking the week of 4th of july.
*** Question about possible user's information leaks in the kibana logs
- Houman: Alex did you see my emails, about leaking user from riemann?
** <2018-06-15 Fri>
- Clojure community
- IROH UI is a thick web application.
- Write a job description for IROH Services Team
** <2018-06-01 Fri>
Topics:
- OPs status status
- UI status update
- naga status update
- certificate expiration
- kibana customer dashboards
- PTO
*** OPS status (AWS outage)
Almost 5pm Mountain Time, AWS had issue.
Respin two nodes on int, two node on test, two on prod.
3 failures to Private CTIA, just after 5pm, for 5 minutes intermitendly.
Craig: that's awesome, only 5min.
The network outage that also had a problem between zones. They didn't have a
downtime for the entier region. Only some racks on some regions.
We just lost 2 instances in prod.
Craig: glad we're minimaly impacted. Does there are any change we should do?
Because there won't much things we could do execpt.
*** UI Status update
Chris:
Daniel working on the demo server.
Fix a couple of bugs.
Find some problems with ATS integration.
The casebook doesn't load at all.
Sometime the menu as layout.
*** Naga Status Update
Paula: Graph API is what I'm working on.
*** Certifcate expiration
See: https://github.com/threatgrid/iroh/issues/1988
Craig: major problem is with refresh-tokens.
*** Kibana dashboard.
See: https://github.com/threatgrid/iroh/issues/1989
*** PTO
- Alex: Thursday / Friday next week
- Paula: Friday next week
- Chris: after cisco live some PTO
- next week in Bosten: Wednesday, Thursday next week
** <2018-05-30 Wed> Secretary: Bart
- Topics: release
** <2018-05-23 Wed>
** <2018-05-14 Mon> Secretary: Jesse
** <2018-04-24 Tue> Secretary: Chris
QA:
** <2018-04-23 Mon> Secretary: Jesse
** <2018-04-11 Wed>
- 1.0 release in Prod
- Thanks to everyone in the team
** <2018-03-27 Tue> Secretary: Daniel
Topics:
- release
- TravisCI build change
*** release
** <2018-03-14 Wed> Secretary: Daniel
- topics: update with PR
- RSA Conf
- #148
** <2018-03-02 Fri> Secretary: Craig
Topics:
- Alex: Int migration
- Alex: Feature request
- Brian: Consumer
- Houman: Release cut
** <2018-02-28 Wed> Secretary: Yann
*** SAML Vulnerability
- Check how we are vulnerable to XML
- throw error when we saw a comment inside an attribute or a value in XML
*** Blocked on ES maximum nb of field error
Alex: Blocking due to maximum nb of fields error.
Guillaume:
We have choices:
1. drop the index on int right now
2. increase the limit
3. wait for migration
Craig:
- fix the mapping by hand
- drop the existing indexes
*** Blocked
Matt:
Another problem, we can't add attack_pattern due to name error.
Will fix tomorrow.
*** Some dev start to become painful
Guillaume:
Working on project is starting to be heavy.
Some changes are becoming really painful.
Maybe rethink the monolith.
Perhaps, refactor some tests.
For example, schema change start to become difficult to handle.
Sometime tests work locally and not on travis.
Perhaps find a way to split the repository.
Craig:
We misse automated test against Integration to have short failure
*** Release status
Houman:
- We should cut a full release.
- Craig: wait for Guillaume, wait until Friday.
*** Cisco AnyConnect Problem
Sam:
- I need to go to Cisco today, I cannot connnect to the VPN.
** <2018-02-27 Tue> Secretary: Paula
*** Report from Berlin
*** Int / Test
** <2018-02-26 Mon> Secretary: Alex
- hour of Airport arrival
- figure out some food. Perhaps barbecue for the next day.
- next day shopping run.
- Turn off non admin user until API key system. Only 3 people.
- private CTIA as a module
TG UI Team on Thursday.
All day hackaton.
Code Freeze last week of march.
*** TG Login
Contact Norman Richard to start working on OpenId Connect Code Grant Flow.
#+BEGIN_SRC
[6:26 PM] Norman Richards: the oidc isn't integrated yet, but you can do non-oidc requests to int now
[6:26 PM] Norman Richards: POST :USER_OAUTH/clients?api_key=:KEY&application_name=app1&description=test&homepage_url=\
:CB&redirect_uri=:CB
[6:26 PM] Norman Richards: That will create on oauth client
[6:27 PM] Norman Richards: :CB is your callback url
[6:27 PM] Norman Richards: sorry - that's not all expanded... So not entirely helpful... Let me get a better example
[6:28 PM] Norman Richards: POST https://int.threatgrid.com/api/v3/users/:yourlogin/oauth/clients?api_key=****&application\
_name=***&description=***&homepage_url=***&redirect_uri=***
[6:30 PM] Norman Richards: To get your clients: GET /api/v3/users/:yourlogin/oauth/clients
[6:30 PM] Norman Richards: /api/v3/oauth/authorizations is the auth endpoint
[6:31 PM] Norman Richards: /api/v3/oauth/token is the token endpoint
[6:31 PM] Norman Richards: /api/v3/oauth/refresh is the refresh endpoint
[6:31 PM] Yann Esposito: there is a refresh endpoint?
[6:31 PM] Norman Richards: You won't need it for oidc though
[6:32 PM] Norman Richards: Though you could use it if you wanted
[6:32 PM] Yann Esposito: because just finished to read the OAuth2 RFC, the refresh and token are the same endpoint for us
[6:32 PM] Norman Richards: You'll be able to do everything at the auth endpoint, but I need to merge the oidc branch to set the "openid" scope and id token support in
[6:32 PM] Norman Richards: Yes - I would prefer they were the same
[6:32 PM] Yann Esposito: ah ok
[6:33 PM] Norman Richards: But they aren't, because that's the way it was initially implemented
[6:33 PM] Norman Richards: I might merge them all together and leave the refresh-only endpoint for just meraki
[6:34 PM] Yann Esposito: ok thanks a lot, I think I could start to work with that
[6:34 PM] Yann Esposito: Craig asked me to use the code grant workflow
[6:34 PM] Norman Richards: Basically, we had a legacy oauth2 implementation from last year for the meraki integration, and there were some unfortunate design decisions made. I'm considering just making new endpoints that work the way I think they should
[6:34 PM] Norman Richards: yes - so you'll need /authorizations and /token then
[6:34 PM] Yann Esposito: I didn't digged must in OIDC yet
[6:35 PM] Norman Richards: It's exactly like what you are used to except you use scope "openid"
[6:35 PM] Yann Esposito: but I just finished coding the code grant workflow
[6:35 PM] Norman Richards: and you get back an id_token on /token
[6:35 PM] Yann Esposito: ah ok! Perfect :)
[6:35 PM] Norman Richards: when you exchange the code
#+END_SRC
*** Tenzin Conf
*** Offsite
1st day. relax together, couple organized presentation.
Architecture overview.
Paula can bring an Apple TV
Monday night, barbecue.
Thursday night, cookie out.
** <2018-02-23 Fri>
** <2018-02-19 Mon> Secretary: Jesse
Guillaume PR
Yann: OAuth2 PR
Paula: CTIA
Wednesday working remotely
Jesse: dep update, new features in cljs
Matt:
- working on graphql for TTPs entities,
- might split graphql tests
- lot of copy/paste in CTIA.
** <2018-02-07 Wed> Secretary: Yann
*** wait_for for refresh in ES impact
We can have up to 10s wait per bundle.
Craig: Can we define what the sets are? By type or dependencies.
Matt: by type.
Craig: wait_for is basically flushing the index.
#idea: We could use a pmap, except for relationships.
- Craig: What is the actual impact we're seeing in time?
- Matt: About 10s.
- Alex: We will have the same problem on all endpoint.
- Craig: we should only restrict the wait_for that endpoint not for all other
routes.
- Alex: I think we can wait to parallelize. I'll try to merge myself the
bundles.
- Craig: at this point this is not a real problem.
Perhaps when we start importing from other systems.
Another option is to update iroh-collect.
#action: restrict the wait_for for the bundle route only
#opional: parallelize the inserts
*** Chris: stuck too long on #1225
Chris: I been stuck way too much time. I must pair with somebody on this.
Craig: Two things:
1. getting familiar with the graph.
2. we need to give you a crash course to our data model.
I think Jesse can walk you through the graph. Let me know when you'll be able to
do that then I'll come to the end of the meeting when I can later today.
#action: pair on https://github.com/threatgrid/iroh/issues/1225
*** Release status
- 3 blockers yesterday. Still one blocker.
- Paula: we could withdraw the module name change, that is not really part of the fix.
- Craig: yes.
#action: get rid of module name change.
Lack Guillaume, Sam, Brian, Daniel for Customer call
** <2018-02-05 Mon> Secretary: Chris
- 3 blockers:
** <2018-02-01 Thu> Secretary: Alex
Topics:
- Chris going to Vegas office from Monday to Wednesday
- Paula? deps, almost
keep this short.
We might droping AMP in the front of our name.
Users and module configuration.
Pb with sharing snapshots.
** <2018-01-31 Wed> Secretary: Matt
Report for the call:
- 6 people want to integrate and push sightings to our private store.
- we should go to phase B of OAuth2
- very solid bundler and make async work
- Crying to get more heads
- For Matt we need to change the name and get all next gen FW events
- More work on translating events.
Apparently Visibility UI will have major changes.
*** Spectre patches
Craig:
Priority for you and Marvin. So at your convenience.
I would prefer to impact production during the evening.
*** CTIA Investigate Issue
Paula should give a hand to fix that.
It bugs on test too.
*** HTML route
@GB show me some code in CTIA that does just that.
I'll try it tomorrow.
--
Craig: Happy with current momentum.
** <2018-01-30 Tue> Secretary: Brian
*** Scratchpad Service (Guillaume)
Guillaume:
Like our investigation, but more structured like a Gist
We use the CTIM entities only instead of raw text.
Drag & Drop observables and/or entities
Craig:
Will make buckets.
Next step will be to make a way to collect material.
We want people to put labels.
Cross product / UI elements / services.
Chris: Does this will replace Snapshots?
Craig: no.
These are just a bucket of objects.
You could turn into an investigation, an incident, or share it.
*** Error Reporting in IROH UI Interface (Jesse)
Coordinate to check error data structure returned.
*** AMP Visibility design recap
Kind of a design bottleneck.
When it goes in the design column.
They can review them.
So they can leave it to the hands of the dev.
*** AMP Visibility Builds
Going from in-progress to QA and to Delivered
Should be different from the rest of the product to deliver faster.
Shorter loops.
Figure out the communication model for that.
There should be a build for every commit in master.
Houman: don't move open ticked to the QA
** <2018-01-29 Mon> Secretary: Houman
*** Status of The release
**** Daniel
- 38 PRs
- Deployment work well for UI
- IROH Ops
**** Guillaume
- Scratchpads
- next step should be a revision system based on CouchDB ideas
**** Houman
- working on release candidate
**** Sam
- Working on design of visibility
- uploading notes from last week meeting
**** Yann
- OAuth2 provider
- Work close the RFC for the workflow
- A part of the workflow isn't specified in the RFC of OAuth2
- To the point to write code to construct the tokens as signed JWTs.
**** Paula
- basic update have been identified
- @Craig: revert to clojure-1.8
- @Craig: get you back working on the AMP Global Intel Board
**** Matt
- Working on AMP Investigate module to report the progress when timeouts occurs
- I report the number of computer that have been enriched and the number of
files that have been checked.
- I report the timeout to riemann
- The AMP module is becoming very big. We'll need to refactor.
- @Craig: next changes:
+ summary
+ event types
- @Craig: this module should be replaced by push from AMP
**** Jesse
- working on the board
**** Chris
- issue for improving snapshots
- @Craig going back to 1732, the slowness one
- I'm pretty sure its slow because advanced is turned off
- we have a way to reproduce it
**** Alex
- Hydrant is updated to use latest CTIM
*** Investigation & Snapshots / Incident & Scratchpads
*** UI rewrite?
** <2018-01-25 Thu> Secretary:
Daniel: Everybody with ops experience should check AWS docs
- https://github.com/threatgrid/tenzin/wiki/Draft-::-EU-and-AJPC-AWS-Regions-Plan
- https://github.com/threatgrid/tenzin/wiki/AWS-and-Operations-Redesign-Proposal-::-Phase-1
Craig will be out of office after 3pm.
** <2018-01-24 Wed> Secretary: Paula
*** Timeout issue
- AMP enrichment timeouts
- module specific
Lets get him some numbers and pass it to AMP for them to do their work.
We need to track and record the timeouts.
We need to give Charles timeout charts.
Generatic a sightings records, here are the 50 targets.
Caching, etc...
*** Client Lib, Visibility Lib
With TG integration, we need them to use our UI and API.
Chris put together a PR about how do we break this up.
Where we break things appart?
It's actually the investigation engine.
Its the first things we should ship.
Reusable widgets they are most of them will be pure JS
*** Deadline
For RSA April 16th, GA, so remove the β.
We'll become a real product.
Pretty aggressive timeline.
- 17 things we need to take care of.
- TG Integration
- Visibility UI
- More relation graph
- filtering
- better arrow
- solving locality, scroll down, and not lose context
- CSRF for response API
- Enrichement API work
- JWT security review
- storage limits and rate limiting
*** Metrics we need in production
- risk, socket exhaustion
- jetty server
- number of open socket, threads, workers, etc...
- how to monitor ES clusters
** <2018-01-22 Mon> Secretary: Guillaume
*** Contact to TG Integration
Need to be tracked an recorded.
Daily update with Alex B.
Really tight relationship with TG.
Jesse will your contact, Guillaume other contact.
*** AMP Global Intel Next Step
- Matt Review
- CAPEC (aggregating our ETS)
- Work on mappings
Dean is demoing Visibility to some big client.
He written a texto: Visibility kick ass, we won.
*** IROH-UI
Thing very simple.
** <2018-01-18 Thu> Secretary: Jesse
- topics: nrepl in ctia
- TG mapping
** <2018-01-17 Wed> Secretary: Alex
** <2018-01-16 Tue> Secretary: Yann
*** Prod Patching Meltdown
Daniel:
- patched the kernel
- Plan on doing the patch one by one and then moving on
Alex: You already did this on INT?
Daniel: everthing looked fine to do the same on prod to be able not to have any
downtime.
Daniel also remarked that we will certainly do the same when the patch for
spectre will come out.
*** TG indicator
Alex: Adding mappings into the TG indicator to the metadatas of Mitre Att@ck
It won't go well because the function is neither surjective nor injective.
There is no way to automate it.
Confident to get some very good mapping. For example, powershell scripts.
Will do another 10 indicators as an example.
Matt: we're be able to automate it after the manual work?
Alex: Of course that the goal.
I'm building a PoC
Alex: Made a PR on hydrant. Everybody welcome to review.
Jesse: I have some PR to review on IROH repo.
Matt: already done two.
** <2018-01-11 Thu> Secretary: Chris
*** Story boards
AMP Global Intel: Alex should take care about this
Visibility one
** <2018-01-10 Wed> Secretary: Sam
*** status update
**** Paula:
I pushed to CTIA everything is up to date except Clojure.
Because CTIA use CTIM.
Craig: what have you learn in this process?
Paula: Pick exclusions in deps, nil return now a 500 instead of 404, added a
middleware to replace 500 by 404.
Matt: I added a comment to your PR
Paula: I went through a lot of the wrappers.
Matt: what about clj-momo
Paula: flanders is done
Craig: One strategy for clj-momo is splitting it appart.
**** Alex
It's freezing.
Yesterday wrapped up theado, zeus, esl blocklist.
Still in progress, really close to put under review.
Example entries, data processing.
I make judgements.
**** Daniel
Patches to the kernels.
** <2018-01-09 Tue>
*** offsite
Approved, not sure about the budget.
Big mantion or something.
The first full week that match.
Maybe next time to Nice.
Austin to F1 then Monaco.
Big house and hangout for the week.
TG UI team is in Austin
Ops team is in Austin
*** yesterday meeting
SBG Architects Snohole and Dede.
Its good, we have their faith in term of model.
They described the IROH Architecture.
We're figuring out how to make our plan big.
It's pretty cool, we gain confidence in leadership of SPG.
Brian comment: that's awesome.
I have confidence in their vision.
But what I do see, accross the SPG there some replication.
All of the ingineering will be realigned.
It was really cool.
It's likely lead to more resources.
We're not gonna build it with just our toolchains.
We're still in a story telling phase.
Certainly high volumes for tomorrow.
We're gonna be in there probably by the end of this year.
GA: Which mean all AMP users and TG users will have access to AMP Visibility.
If you look at the IROH Roadmap.
Very few feature that are GA gateways.
Probably starting the end of the year.
In June or July do even more work on next gen firewall in Barcelona.
*** update UI stuff
So Jesse working on the timeline stuff.
Bugs from the beta.
I don't have time to triage and look at details in the UI.
*** Job description
Updated job description.
Our requirement really in Clojure.
** <2018-01-08 Mon>
@Alex: I'm working on splitting up the judgements for both domains and IPs.
@Matt: ask what is the size of the bundles
@Alex: not so big, its small enough to do everything on memory
@Alex: theado, bankin trojans, the feed split in theado and theadog/f?
@Matt: should we add external-id to our relationships
** <2018-01-05 Fri>
- Paula worked on deps
- Braintree t-shirt
- deps discussions, for example about CTIM
*** Secretary @Alex
- @Chris UI breakout:
- @Craig: Show the new board
- Offsite
*** UI breakout
- talk with Guillaume and Jesse
- separate repo
- CI for each branch (to be ok not to have a million builds)
- decouple js modules from cljs to be able to move to modules that could be included
- Jesse & Craig question: why is separating js is not a separated concern from
moving to another repo
- @Craig:
- phase 1, let first make another repo (ASAP today)
- phase 2, builds and deployed to UI Staging
- phase 3, how are we going to break out our lib into widgets
*** Project Board(s)
@Craig
- Started building the roadmap
- IROH Roadmap: High level stuff, org level project so can take issues from any
of our repos
- AMP Global Intel: data importers, tenzin, etc...
- IROH Services: CTIA + IROH APIs services
- AMP Visibility: UI
- IROH Operations: for Ops
Will try to go to github certified instead of private github.
Will take care of populating those.
Chris: is there a way to see all the boards?
Craig: No, its a limitation from gh project
*** Offsite
- Austin because many people are there
- Ranting a big villa
- sharing of the room
- max 15 people + a few guest
- Purpose:
- hangout for 3 days,
- cook a couple of meals
- pool
- naturally a lot of discussion going on
- day trips, etc...
- Feb 26 -> March 3rd in the worst case
- Arrange one group dinner
*** New position
- candidate anywhere on our team, UI vs backend.
- prefer UI focus, or at least UI capable
- same level as experience as us, no entry level position
- prefer women or minorities, let's reach out
** <2018-01-04 Thu>
- TG Integration, iroh-auth side work by gb and yann
- Design coordination for 20 minutes
- ui staging server check (iroh-auth doesn't respond)
- =transient= or =external= resolved by an external ref lookup
- @Paula dependencies?
- @Chris component listing? slow, add more things to the cljs
Notes from Paula:
Topics
- Threatgrid integration and timeline for that
- UI Staging server.
Is it working? Jesse asked. Matt encountered this morning. Are any IROH Auth paths working?
Craig and Brian identified more issues during demo with Mike Auger
Integration with TG. Work on IROH Auth services to be done by Guillaume and Yann
Work on TG side to support us with user’s credentials.
Meeting with Face/mask team at 12:30 (API and UI team)
This has a high priority because of the RSA conference. We want to accomplish a
working environment for the TG UI team ASAP.
Action Item: Daniel fixing UI staging server fixed, and getting resources for
that.
Comments by Daniel: It’s just IROH Auth. Looking further. (Craig: Should be
pointing to INT Iroh Auth)
Updates:
Alex: done with abuse.ch and others. Writing bundlers. Code for Ransomware and
ssl block list is already in github.
Paula: updates triaged, and minor updates to be checked in soon.
Chris update on UI (sound dropped)
On the UI side: Issues were identified on public beta. There may be more design
work, particularly around inspector. How to get summary of sightings for
observables? How make UI more reactive really large screens? Ideas from Sam.
** <2018-01-03 Wed>
@Paula:
- Most of our deps can be updated.
- compojure from schema to spec
- we should use more spec, but we should check first that its ok
- its more verbose and harder and can be a rabbit hole
- @craig: def swagger schema
- everybody should use more spec, and jump to 1.9
Talk with Guillaume and Yann, to use Auth, OpenId Connect to be able
to make TG user to login in Visibility.
** <2018-01-02 Tue>
*** Craig
Happy New Year, personnaly had great 3 week vacation
**** Change the format of the daily, more formalization
Daily standup don't make much change because of TZ.
But important to have Face to Face to talk about more informal topics.
Talk about things not necessarily captures in github issue.
- Going to assign a "secretary" keep tracking the suggested topics and write it
to hipchat.
- Select a topic for discussion.
- Must be fixed in 15min.
- Rate the meeting between 1 and 5.
- Identified topics.
Series of topics for lead by example.
**** priorities for Coming Quarter
- Process Simplification
- replace waffle with github project
- move to git-flow
- Semantic Versionning
- Team Offsite
- thinking mid-late February (probably Austin)
- not be a working off-site, brainstorming, socialization
- Public Beta Program: Supporting Users (lot of time on the phone)
- RSA Conference, early April, need to make some kind of announcement
- TG/AMP/Visibility Integration
- Shared Incident Response Services
- scratchpad, as a way for products having an Incident response notebook
- Incident definition
- CoA
- SBG Architecture Demo
- Undefined, integration outside of ATS, how do I get data from my firewall to
Visibility
**** One Goal/Perspective for your Carreer this Year
***** Jesse: Linear Algebra
***** Frank: Make a good mock for AMP
***** Matthieu: gain expertise in security business domain
***** GB: focus on stream computing, nix, mesos, CQRS/ES, architecture that scale
***** Yann: unification of stream data structure
***** Paula: be more involved with the process
***** Alex: leadership, build tools
***** Houman: gain deeper knowledge in security
***** Daniel: I was thinking to learn more about golang
***** Chris: read 12 books, one of them on security, and apply more to speak
***** Brian: I want to check in code, I can contribute on the webmonkey side
***** Craig: life/work balance issue, minimize micro-management
- pay attention into the things that are a bit further ahead,
- larger scale manager/architect role
- path (PE principal Engineer) individual contributor / director ingineering
** <2017-12-04>
- Tomorrow will be our open Beta
- check all PR to merge before the RC
- remove ie11 support temporarily
** <2017-11-29>
- Houmann is confident
- two blockers, we should be safe
- Tomorrow last day of Steve
- Craig: @Paula how goes the hack on AV Signatures? → link to PR with commit 1e7e0ba in hydrant.
** <2017-11-28>
Craig is sick, Barb too.
Relase candidate cut, we are release feature complete and complete edge goal too.
Cut a release on the original target date. Not too bad.
Life for the QA easier.
Check that we describe properly QA test.
All Epics should be under "Epic Review".
Release notes.
Test the app and file issues by milestoning them with Public Beta
** <2017-10-27>
- co-working space is cold (50°F) (Sam + Chris + Brian)
- Craig fait semblant de freeze, mais ça se voit
** <2017-10-26>
- Craig:
* a new cross platform integration named DD
* we already do 3 points out of their high level strategy goals
* there's DD and Sne'Hal?
* Two demos to the ESA team
* --- lost conn for 1 min... fuck...
* We want ESA to send us all their sightings
* We need to try to get TG users to our system
* We need to come up with an estimate how much is gonna cost to run a private CTIA per customers
* We're gonna pickup speed very quickly in November
* More positive feedback
* next writing up papers and presentations
- Brian/Chris/Sam: I've got some feedback, did you heard them?
- Sam: yes
- Yann: move to API Keys gen
- We're must be sure private CTIA will cost too much
- Paula: all set on the AV stuff?
- Alex is very enthousiastic to grab it all the sources of informations
- Dean think that importing 3rd party will have big impact (Craig disagree with that)
- After next year, everybody will be on us. Sales, features, etc...
** <2017-10-18> - nil
** <2017-10-17> - nil
** <2017-10-16>
- execute with other teams
- Plan together, if you are blocked on the design, be pro-active
- @chris: back from the Conj, I'm ready to go, I learn cool things from conj
- next few weeks will be ui everyday
- type clojure can write test and specs
- generate specs from a vector of "edn"s
- btw if you're blocked on the investigation model
- @jesse worked a bit on that on friday
- @craig I've got minimum time, don't block on me on style questions
- @craig:
- like ctim objects, UUID, Title, short/long description, sourceable object (where they come from)
- all bunch of other data
- observables, to be searchable
- they're immutable
- default permissions for private CTIA
- @craig to @Brian: how we want to present them. How we do a search?
- @brian: just a table with titles, names, descriptions.
- @craig how do we show when they have the same name.
- @craig for the user, it should understand to understand it is a "snapshot"
** <2017-09-26>
- taking time, take care of yourself
- make time were communicating with your co-worker slowly
- make sure you're in the right mindset
- We need to continue to grow, we're really close to being a production system.
- Driving the architecture and platform story.
- We need to make sure we focus on the problems of the customers
- We'll have a lot of exposure to the real world
- The bar is pretty low for us
- Take the work and organize it and present to the user with established
technique.
- Half, making great doc, and review, and had to keep an open mind
- New PR are changing a lot and are little intimidating
- we have to assume the best from the rest of the team
- Don't write the comment that can upset you. So reach out and webex.
Any blockers?
- @Paula: I've got to reach out about some details about TTPs
- @Alex: helped Guillaume and Nargol, updating hydrant
What we want private CTIA to be for the rest, most event should go through it?
With the public ß, people could inspect it.
Should be a data store for incident response flow.
AMP and incident model, major step for design work.
Next iteration on incident target.
Happy with everyone perf.
This team has made that real.
Any blockers? Answer on something?
Share some stuff.
We could collectively increase our knowledge.
Clojure and Java one time inspection.
Increasing perf in Java.
I want us to really focus on starting from empirical data. Having a good
understanding on the perf will be necessary to reach better quality.
We're talking about doing java concurrency.
That are important for clojure dev.
Somebody can add to the list, and brown?
@Paula: idiomatic clojure, from the cognitect people
It can be difficult to learn from them if you aren't following certain people.
Like the seq abstraction.
- transients...
Improvement of common knowledge of clojure runtime.
- schema then protocols, and records, etc...
- it seems of overly complex way to deal with data structures.
- spec instead of schema?
@Nargol, I'll try to be online because I'm at the symposium.
@Craig, take the buffet :)
Reference in a new issue View git blame Copy permalink