diff --git a/.gitignore b/.gitignore index 9912f513..2cde0486 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,4 @@ ltximg HWP/ .*.icloud +.stack-work/ diff --git a/Cisco.org.gpg b/Cisco.org.gpg index 4da93194..80535b16 100644 Binary files a/Cisco.org.gpg and b/Cisco.org.gpg differ diff --git a/Cisco.org.gpg_archive b/Cisco.org.gpg_archive index ff26bf5e..b1c2c3bb 100644 --- a/Cisco.org.gpg_archive +++ b/Cisco.org.gpg_archive @@ -2945,3 +2945,468 @@ Improvement of common knowledge of clojure runtime. @Nargol, I'll try to be online because I'm at the symposium. @Craig, take the buffet :) + +* Tech notes + :PROPERTIES: + :ARCHIVE_TIME: 2019-04-04 Thu 16:27 + :ARCHIVE_FILE: ~/.deft/Cisco.org.gpg + :ARCHIVE_CATEGORY: Cisco.org + :END: +** OPS Connect to Int + +To find the IPs: + +https://tg-iroh.signin.aws.amazon.com/console + +Last time IP of =tenzin.int.iroh.site=: 54.165.154.145 + +Then + + #+BEGIN_SRC + ssh -i ~/.ssh/tenzin_master_int -o IdentitiesOnly=yes ubuntu@tenzin.int.iroh.site + sudo su - + salt '*iroh-01*' cmd.run "ifconfig" + # GET THE IP + sudo su - + ssh -i /etc/salt/tenzin_master ubuntu@XXX + sudo su - + #+END_SRC + + And you could do: + + #+BEGIN_SRC + cd /srv/iroh; ls -lath + service iroh restart + lsof -nP | grep LISTEN + #+END_SRC + +* Email of users + :PROPERTIES: + :ARCHIVE_TIME: 2019-04-04 Thu 16:29 + :ARCHIVE_FILE: ~/.deft/Cisco.org.gpg + :ARCHIVE_OLPATH: Epics + :ARCHIVE_CATEGORY: Cisco.org + :END: + +** channel: email + +Hi Eduardo, + +In order for everyone to be aware of the situation here is a short resume: + +1. Since we're using the IDB, new user records don't not contain an email field in the profile. + The IDB does not return that data as specified in the OIDC protocol (see my remark). +2. I stated the problem to the IDB team +3. In order to mitigate that, for AMP accounts, I luckily found a workaround using a non + standard field. Thus, currently, new users and existing users who login via AMP + should have an email set in our DB. +4. I asked the IDB team again to provide us the email during login also for TG users. + +The current status: + +Our DB should start to be filled with emails for all user that login to CTR. +Existing user that do not login won't have their email set. + +If the IDB team find a way to update their configuration to pass down the email +information for TG user, the DB might also be filled automatically without +work to be done by the CTR team. + +> + Adding Snehal, Craig, and Guillaume for awareness +> +> Hi Yann, +> +> Could you please provide an update on the issues 2440 and 2504 around not +> getting user emails? There are 1548 user ids from AMP customers with no email +> associated. +> +> Something seems to have changed, since recently we’ve been getting no emails for +> new users. This is essential for us to track adoption by product and set up +> targeted email campaigns. +> +> Thanks + + +* History API via Event Store :TOC_3_gh:QUOTE: + :PROPERTIES: + :ARCHIVE_TIME: 2019-04-04 Thu 16:29 + :ARCHIVE_FILE: ~/.deft/Cisco.org.gpg + :ARCHIVE_OLPATH: Epics + :ARCHIVE_CATEGORY: Cisco.org + :END: +#+BEGIN_QUOTE +- [[#oauth2-provider][OAuth2 Provider]] + - [[#workflow][Workflow]] + - [[#authorize][=/authorize=]] + - [[#approve][=/approve=]] + - [[#refuse][=/refuse=]] + - [[#token][=/token=]] + - [[#oauth2-in-iroh-auth-spec-rfc-second-pass][OAuth2 in IROH-Auth Spec RFC second pass]] + - [[#vocabulary][Vocabulary]] + - [[#client-registration][Client Registration]] + - [[#protocol-endpoints][Protocol Endpoints]] + - [[#obtaining-authorization][Obtaining Authorization]] + - [[#oauth2-provider-epic][OAuth2 Provider Epic]] + - [[#functional-spec][Functional Spec]] + - [[#tasks][Tasks]] + - [[#technical-spec][Technical Spec]] + - [[#oauth2-epics-3rd-pass][OAuth2 Epics (3rd pass)]] + - [[#spa-compatible-oauth2][SPA compatible OAuth2]] + - [[#users-made-oauth2-clients][User's made OAuth2 clients]] + - [[#internal-user-representation][Internal User Representation]] + - [[#oauth2-client-credentials-grant][OAuth2 Client Credentials Grant]] + - [[#iroh-admin-dashboard][IROH Admin Dashboard]] + - [[#oauth2-enhancements][OAuth2 Enhancements]] +- [[#scopes-dictionary][Scopes Dictionary]] +- [[#document-for-raghavaiah][Document for Raghavaiah]] + - [[#franks-proposal-auth-config-untangling][Frank's proposal; Auth config untangling]] + - [[#int][INT]] + - [[#test][TEST]] + - [[#proposal][Proposal]] + - [[#int-1][INT]] + - [[#prod-nam][PROD NAM]] + - [[#prod-eu][PROD EU]] + - [[#prod-apjc][PROD APJC]] + - [[#test-1][TEST]] +- [[#daily-standup-meeting][Daily Standup Meeting]] + - [[#2019-02-27-wed][<2019-02-27 Wed>]] + - [[#release-119][release 1.19]] + - [[#individual-updates][individual updates]] + - [[#uiux][UI/UX]] + - [[#misc][Misc]] + - [[#2019-02-25-mon][<2019-02-25 Mon>]] + - [[#2019-01-18-fri][<2019-01-18 Fri>]] + - [[#individual-update][Individual update]] + - [[#2019-01-23-wed][<2019-01-23 Wed>]] + - [[#ops][ops]] + - [[#rel-116][rel 1.16]] + - [[#individual-update-1][Individual update]] + - [[#ux-design-update][UX Design update]] + - [[#design-update][Design update]] + - [[#2019-01-11-fri][<2019-01-11 Fri>]] + - [[#ops-weather-report][Ops Weather Report]] + - [[#individual-reports][Individual Reports]] + - [[#design][Design]] + - [[#2019-01-09-wed][<2019-01-09 Wed>]] + - [[#ops-1][Ops]] + - [[#reports][Reports]] + - [[#2019-01-08-tue][<2019-01-08 Tue>]] + - [[#ops-2][OPS]] + - [[#release-status][Release Status]] + - [[#angela][Angela]] + - [[#2019-01-04-fri][<2019-01-04 Fri>]] + - [[#2018-12-07-fri][<2018-12-07 Fri>]] + - [[#topics][Topics]] + - [[#ops-3][Ops]] + - [[#rel][Rel]] + - [[#2018-11-28-wed][<2018-11-28 Wed>]] + - [[#release-114][release 1.14]] + - [[#ops-report][Ops report]] + - [[#individual-report][Individual Report]] + - [[#ux-design-report][UX Design Report]] + - [[#2018-11-26-mon][<2018-11-26 Mon>]] + - [[#2019-01-11-fri-1][<2019-01-11 Fri>]] + - [[#2018-11-19-mon][<2018-11-19 Mon>]] + - [[#ops-4][Ops]] + - [[#ui][UI]] + - [[#individual-reports-1][Individual reports]] + - [[#jwt-lifetime][JWT lifetime]] + - [[#2018-10-31-wed][<2018-10-31 Wed>]] + - [[#individual-reports-2][Individual Reports]] + - [[#2018-10-22-mon][<2018-10-22 Mon>]] + - [[#individual-updates-1][Individual Updates]] + - [[#2018-10-10-wed][<2018-10-10 Wed>]] + - [[#2018-10-01-mon][<2018-10-01 Mon>]] + - [[#2018-09-19-wed][<2018-09-19 Wed>]] + - [[#ops-weather-reports][Ops weather reports]] + - [[#release-report][Release report]] + - [[#doc-report][Doc report]] + - [[#personal-report][Personal report]] + - [[#ux-update][UX Update]] + - [[#pto][PTO]] + - [[#2018-09-18-tue][<2018-09-18 Tue>]] + - [[#2018-09-11-tue-secretary-nola][<2018-09-11 Tue> Secretary: Nola]] + - [[#topics-1][Topics]] + - [[#pr-merge][PR merge]] + - [[#pto-1][PTO]] + - [[#2018-09-10-mon][<2018-09-10 Mon>]] + - [[#yann][Yann]] + - [[#ux-design][UX Design]] + - [[#idp-issue][IdP Issue]] + - [[#2018-08-14-tue][<2018-08-14 Tue>]] + - [[#2018-08-17-fri][<2018-08-17 Fri>]] + - [[#topics-2][Topics]] + - [[#ops-report-1][Ops report]] + - [[#individual-reports-3][Individual Reports]] + - [[#ux-design-update-1][UX Design Update]] + - [[#pto-2][PTO]] + - [[#2018-08-13-mon-secretary-paula][<2018-08-13 Mon> Secretary Paula]] + - [[#individual-report-1][Individual report]] + - [[#2018-08-08-wed-secretary-john][<2018-08-08 Wed> Secretary John]] + - [[#individual-report-2][Individual Report]] + - [[#2018-08-07-tue][<2018-08-07 Tue>]] + - [[#brian][Brian]] + - [[#2018-08-06-mon][<2018-08-06 Mon>]] + - [[#bart-resigned-by-mail][Bart resigned (by mail)]] + - [[#individual-report-3][Individual report]] + - [[#2018-08-03-fri][<2018-08-03 Fri>]] + - [[#topic][Topic]] + - [[#individual-update-2][Individual Update]] + - [[#ops-report-2][Ops Report]] + - [[#release-status-1][Release Status]] + - [[#2018-08-01-wed][<2018-08-01 Wed>]] + - [[#topics-3][Topics]] + - [[#update-from-craig][Update from Craig]] + - [[#individual-reports-4][Individual Reports]] + - [[#blocking-bug-622][Blocking Bug #622]] + - [[#ops-5][Ops]] + - [[#pto-3][PTO]] + - [[#2018-07-25-wed-secretary-matt][<2018-07-25 Wed> Secretary Matt]] + - [[#individual-reports-5][Individual Reports]] + - [[#2018-07-24-tue][<2018-07-24 Tue>]] + - [[#release][Release]] + - [[#operation-report][Operation Report]] + - [[#release-notes][Release Notes]] + - [[#2018-07-23-mon-secretary-alex][<2018-07-23 Mon> Secretary Alex]] + - [[#individual-reports-6][Individual Reports]] + - [[#2018-07-17-tue][<2018-07-17 Tue>]] + - [[#topics-4][Topics]] + - [[#2018-07-16-mon][<2018-07-16 Mon>]] + - [[#topics-5][Topics]] + - [[#individual-reports-7][Individual Reports]] + - [[#2018-07-13-fri][<2018-07-13 Fri>]] + - [[#2018-07-11-wed-secretary-bart][<2018-07-11 Wed> Secretary: Bart]] + - [[#individual-update-3][Individual update]] + - [[#release-deployment-day][Release Deployment Day]] + - [[#conversation-in-the-iroh-channel][conversation in the IROH channel]] + - [[#soliciting-topic][soliciting topic]] + - [[#impersonate-api][impersonate API]] + - [[#integrating-with-amp-test-env][integrating with AMP test env]] + - [[#update-from-john][update from John]] + - [[#2018-07-09-mon][<2018-07-09 Mon>]] + - [[#release-1][Release]] + - [[#pto-4][PTO]] + - [[#][...]] + - [[#2018-07-06-fri-secretary-jesse][<2018-07-06 Fri> Secretary Jesse]] + - [[#release-2][Release]] + - [[#discovery][Discovery]] + - [[#bart-demo][Bart Demo]] + - [[#upcoming-pto][Upcoming PTO]] + - [[#2018-07-04-wed][<2018-07-04 Wed>]] + - [[#2018-07-03-tue][<2018-07-03 Tue>]] + - [[#2018-06-29-fri][<2018-06-29 Fri>]] + - [[#iroh-ui][IROH UI]] + - [[#pto-5][PTO]] + - [[#2018-06-22-fri-secretary-yann][<2018-06-22 Fri> Secretary Yann]] + - [[#release-status-2][Release Status]] + - [[#ops-report-3][Ops Report]] + - [[#ttp-headnodes][TTP Headnodes]] + - [[#pto-update][PTO update]] + - [[#question-about-possible-users-information-leaks-in-the-kibana-logs][Question about possible user's information leaks in the kibana logs]] + - [[#2018-06-15-fri][<2018-06-15 Fri>]] + - [[#2018-06-01-fri][<2018-06-01 Fri>]] + - [[#ops-status-aws-outage][OPS status (AWS outage)]] + - [[#ui-status-update][UI Status update]] + - [[#naga-status-update][Naga Status Update]] + - [[#certifcate-expiration][Certifcate expiration]] + - [[#kibana-dashboard][Kibana dashboard.]] + - [[#pto-6][PTO]] + - [[#2018-05-30-wed-secretary-bart][<2018-05-30 Wed> Secretary: Bart]] + - [[#2018-05-23-wed][<2018-05-23 Wed>]] + - [[#2018-05-14-mon-secretary-jesse][<2018-05-14 Mon> Secretary: Jesse]] + - [[#2018-04-24-tue-secretary-chris][<2018-04-24 Tue> Secretary: Chris]] + - [[#2018-04-23-mon-secretary-jesse][<2018-04-23 Mon> Secretary: Jesse]] + - [[#2018-04-11-wed][<2018-04-11 Wed>]] + - [[#2018-03-27-tue-secretary-daniel][<2018-03-27 Tue> Secretary: Daniel]] + - [[#release-3][release]] + - [[#2018-03-14-wed-secretary-daniel][<2018-03-14 Wed> Secretary: Daniel]] + - [[#2018-03-02-fri-secretary-craig][<2018-03-02 Fri> Secretary: Craig]] + - [[#2018-02-28-wed-secretary-yann][<2018-02-28 Wed> Secretary: Yann]] + - [[#saml-vulnerability][SAML Vulnerability]] + - [[#blocked-on-es-maximum-nb-of-field-error][Blocked on ES maximum nb of field error]] + - [[#blocked][Blocked]] + - [[#some-dev-start-to-become-painful][Some dev start to become painful]] + - [[#release-status-3][Release status]] + - [[#cisco-anyconnect-problem][Cisco AnyConnect Problem]] + - [[#2018-02-27-tue-secretary-paula][<2018-02-27 Tue> Secretary: Paula]] + - [[#report-from-berlin][Report from Berlin]] + - [[#int--test][Int / Test]] + - [[#2018-02-26-mon-secretary-alex][<2018-02-26 Mon> Secretary: Alex]] + - [[#tg-login][TG Login]] + - [[#tenzin-conf][Tenzin Conf]] + - [[#offsite][Offsite]] + - [[#2018-02-23-fri][<2018-02-23 Fri>]] + - [[#2018-02-19-mon-secretary-jesse][<2018-02-19 Mon> Secretary: Jesse]] + - [[#2018-02-07-wed-secretary-yann][<2018-02-07 Wed> Secretary: Yann]] + - [[#wait_for-for-refresh-in-es-impact][wait_for for refresh in ES impact]] + - [[#chris-stuck-too-long-on-1225][Chris: stuck too long on #1225]] + - [[#release-status-4][Release status]] + - [[#2018-02-05-mon-secretary-chris][<2018-02-05 Mon> Secretary: Chris]] + - [[#2018-02-01-thu-secretary-alex][<2018-02-01 Thu> Secretary: Alex]] + - [[#2018-01-31-wed-secretary-matt][<2018-01-31 Wed> Secretary: Matt]] + - [[#spectre-patches][Spectre patches]] + - [[#ctia-investigate-issue][CTIA Investigate Issue]] + - [[#html-route][HTML route]] + - [[#2018-01-30-tue-secretary-brian][<2018-01-30 Tue> Secretary: Brian]] + - [[#scratchpad-service-guillaume][Scratchpad Service (Guillaume)]] + - [[#error-reporting-in-iroh-ui-interface-jesse][Error Reporting in IROH UI Interface (Jesse)]] + - [[#amp-visibility-design-recap][AMP Visibility design recap]] + - [[#amp-visibility-builds][AMP Visibility Builds]] + - [[#2018-01-29-mon-secretary-houman][<2018-01-29 Mon> Secretary: Houman]] + - [[#status-of-the-release][Status of The release]] + - [[#investigation--snapshots--incident--scratchpads][Investigation & Snapshots / Incident & Scratchpads]] + - [[#ui-rewrite][UI rewrite?]] + - [[#2018-01-25-thu-secretary][<2018-01-25 Thu> Secretary:]] + - [[#2018-01-24-wed-secretary-paula][<2018-01-24 Wed> Secretary: Paula]] + - [[#timeout-issue][Timeout issue]] + - [[#client-lib-visibility-lib][Client Lib, Visibility Lib]] + - [[#deadline][Deadline]] + - [[#metrics-we-need-in-production][Metrics we need in production]] + - [[#2018-01-22-mon-secretary-guillaume][<2018-01-22 Mon> Secretary: Guillaume]] + - [[#contact-to-tg-integration][Contact to TG Integration]] + - [[#amp-global-intel-next-step][AMP Global Intel Next Step]] + - [[#iroh-ui-1][IROH-UI]] + - [[#2018-01-18-thu-secretary-jesse][<2018-01-18 Thu> Secretary: Jesse]] + - [[#2018-01-17-wed-secretary-alex][<2018-01-17 Wed> Secretary: Alex]] + - [[#2018-01-16-tue-secretary-yann][<2018-01-16 Tue> Secretary: Yann]] + - [[#prod-patching-meltdown][Prod Patching Meltdown]] + - [[#tg-indicator][TG indicator]] + - [[#2018-01-11-thu-secretary-chris][<2018-01-11 Thu> Secretary: Chris]] + - [[#story-boards][Story boards]] + - [[#2018-01-10-wed-secretary-sam][<2018-01-10 Wed> Secretary: Sam]] + - [[#status-update][status update]] + - [[#2018-01-09-tue][<2018-01-09 Tue>]] + - [[#offsite-1][offsite]] + - [[#yesterday-meeting][yesterday meeting]] + - [[#update-ui-stuff][update UI stuff]] + - [[#job-description][Job description]] + - [[#2018-01-08-mon][<2018-01-08 Mon>]] + - [[#2018-01-05-fri][<2018-01-05 Fri>]] + - [[#secretary-alex][Secretary @Alex]] + - [[#ui-breakout][UI breakout]] + - [[#project-boards][Project Board(s)]] + - [[#offsite-2][Offsite]] + - [[#new-position][New position]] + - [[#2018-01-04-thu][<2018-01-04 Thu>]] + - [[#2018-01-03-wed][<2018-01-03 Wed>]] + - [[#2018-01-02-tue][<2018-01-02 Tue>]] + - [[#craig][Craig]] + - [[#2017-12-04][<2017-12-04>]] + - [[#2017-11-29][<2017-11-29>]] + - [[#2017-11-28][<2017-11-28>]] + - [[#2017-10-27][<2017-10-27>]] + - [[#2017-10-26][<2017-10-26>]] + - [[#2017-10-18---nil][<2017-10-18> - nil]] + - [[#2017-10-17---nil][<2017-10-17> - nil]] + - [[#2017-10-16][<2017-10-16>]] + - [[#2017-09-26][<2017-09-26>]] +- [[#tech-notes][Tech notes]] + - [[#ops-connect-to-int][OPS Connect to Int]] +- [[#email-of-users][Email of users]] + - [[#channel-email][channel: email]] + - [[#introduction][Introduction]] + - [[#plan][Plan]] + - [[#part-1][Part 1]] + - [[#part-2][Part 2]] + - [[#technical-details][Technical Details]] +#+END_QUOTE + +** Introduction + + Multiple recent features/changes would be improved by a system to keep + track of history events (see #2370, #2425, #2426). + + - keep track of login dates (right now, we only keep track of the + latest 5 login dates, we also can't keep much meta infos about those + logins like IP addresses, HTTP referrers, etc...) + - Administrative tasks, a new notion of role will give some user the + right to manage resources like other users / other OAuth2 clients. + As such we should keep track of who is doing what and when to + prevent subtle attack and/or errors. + + For now, all these information can be gathered through our internal logging + system. I think this should become an internal API. Typically our customer will + want to know who is the admin that blocked some user and when. + +** Plan + +*** Part 1 + + Keep track of: + + - user logins + - user profile updates + - client updates (who modified the client and when) + + I think we should start with a very small plan first. We could take + advantage of the current events we are sending to Riemann and put those + events (or only part of them) in a searchable store. The advantage is + that we already have a service with a clear and simple API and is used + in many different other services. + + We should then impose stronger constraints to the event format. It + should be beneficial for both internal API and analysis via Kibana (the + team responsible to analyse user behaviour expressed difficulties in + dealing with the lack of common format and missing infos for some + events). + + This approach will also make it easier to simplify the internal + structure of the client object and be able to remove details about the + internal workflow in that object (and also others). Typically, it should + remove the need for many metadatas of the object (updated-by, + approval-status, etc...) + + During this first step the events should correspond to *things that + happened* and as such should be named in past tense (*User Blocked*, + *Client Deleted*, etc...). + +*** Part 2 + +/remark/: Part 2 is out of the scope of this issue and should be its own issue +later. + + I think we should think about not only using "Events" (things that + happened in the past) but also /Commands/ (requests for things to + happen, named with verb in imperative mood). And from there we might + start to create /Aggregates/ (read-only views constructed from events). + Using this architecture, a single events would be able to be used to + provide different views (historical views about a single client, list of + users that modified clients in the past, stats about users, etc...) + +** Technical Details + +- create an Event Store that can also be a CRUDStore to be searchable. I think + it should be backed by PostgreSQL to ease views when we might need to make + joins. +- the =Event= schema should contain an =event-type= this should always be named in + the past tense. (note this structure provide a quite good template for events + structure that could ease the search in kibana in the future) + #+BEGIN_SRC clojure + (s/defschema Event + {:event-type s/Keyword ;; prefer namespaced keywords that should link to a schema + :event-params {s/Any s/Any} ;; some printable hash-map that match the schema linked by event-type + :emiter + (st/optional-keys + {:service s/Keyword ;; should be an service-name + :user-id s/Str ;; should be the user-id of the person responsible for the event + :client-id s/Str ;; if the operation is done via a client provide the client-id + :impersonated-by s/Str ;; if impersonated should provide the user-id of the master user + :user User ;; all gory details about the User + :org Org ;; all gory details about the org + })}) + #+END_SRC + +- update the riemann-reporter service to be able to write to multiple stores and + with the ability to filter on set of event-types. So we could create stores + that could contains only some event-types. Also provide a list of event fields to track. + As riemann event-store will want all the details about an event while event in a searchable + store should not. +- provide another =send-event= method for the =riemann-reporter= service to be able to send + event with all the mandatory infos, take care of removing all user / org infos from the event destined + #+BEGIN_SRC clojure + (s/defn send-event [event-type event-params emiter-infos] :- Event ...) + #+END_SRC +- update the =send-event= call to match the correct format and configure one + searchable store of riemann reporter store to track those events. +- update the =/login= handler not to put login dates infos in the user object. +- update the =/profile= endpoint to use thoses events to display latests login dates. + diff --git a/HWP.org b/HWP.org index 739df5ed..81cdc787 100644 --- a/HWP.org +++ b/HWP.org @@ -1,46 +1,1202 @@ -#+TODO: TODO TO-CLEAN TO-REVIEW | DONE +# Created 2019-04-01 Mon 13:31 #+TITLE: Haskell for the working programmer #+AUTHOR: Yann Esposito -#+EMAIL: yann.esposito@gmail.com +#+TODO: TODO TO-CLEAN TO-REVIEW | DONE #+LANGUAGE: en #+KEYWORDS: haskell #+PROPERTY: header-args :output-dir HWP :mkdirp yes :tangle-mode (identity #o755) +* TO-CLEAN Introduction + +This is somehow a follow-up from Learn Haskell Fast and Hard. +Which was more about being able to /play/ with Haskell than to /work/ with it. + +This is also an experiment. +I'm not sure if it will be as positive as I hope. +This book try to be a good resource to learn Haskell but to speed up the +learning in the first part I'll skip the explanation about why Haskell does +things the way it does. +As a consequence if you don't keep in mind that there is *very good* reasons to +make some things way more difficult in Haskell than in other languages you +might miss the real reason. + +Also don't forget in the beginning you might only see what is more difficult or +harder to achieve in Haskell. +But for each thing harder, keep in mind that there are very difficult things in +other languages that are solved extremely easily in Haskell. +And I personnally believe the things Haskell make easier are essential to reach +the best balance between speed, elegance, safety and pragmatism with regards to +any programming language I ever used before. + +So this book might be a bit raw. +And in fact not really "fun" unfortunately. +But it should be efficient. + +This book is aimed to be one of the fastest way to learn how to be productive +with Haskell. + +Know that there still will be a very long road ahead once this book will be +finished to master Haskell. That should be ok. Even with those basic knowledge, +you should already be more productive in Haskell than in most other programming. + +Modern computing has unfortunately less to do with algorithmic than to create a +mashup of libs and external APIs. +So while learning all the details of Haskell can seems like an impossible challenge. +Learning the necessary skills to be productive shouldn't be that hard. + +What does this book will talk about. + +1. Having a clean and stable dev environment +2. Basic Introduction to the language +3. Professional Project developement workflow +4. Make command line program +5. Use external libraries +6. Handle the filesystem +7. Handle a few DBs +8. Make a basic REST API + +What Haskell can do few other programming language can. + +Ability to put strong constraingt on part of the code. For exemple you can have +confidence in 3rd party functions. You can be certain that there will be NO side +effect. Or you can also ensure that part of you code can only write logs and not +access the DB. We'll technique that will ensure that subpart of the code will +only access the User table in your DB etc... + +Writting parallel and concurrent code because /very/ easy to write. +While this is generally a nightmare in most programming language. + +** TO-CLEAN What does "working programmer" stand for? + +Being able to: + +- create a new working program from scratch, +- work with the filesystem (read/write files/directories), +- work with BDD (SQLite, PostgresSQL, MongoDB, etc...), +- work with network (send/receive HTTP request), +- make a REST API, +- use most libraries (OpenGL, ncurses, etc...) +- write test for your application, +- to deploy your application + +This is more about being an user, consumer from the Haskell community than being +an active contributor. Hopefully the gap won't be hard to pass from user to +contributor. So I'll write a minimal chapter about how to write your own library +and publish it for other developpers. + +** TO-CLEAN Prerequiste + +The target audience I'm writting this book for is software developpers. + +You should: + +- be familiar with some programming language, +- be familiar with command line in a shell, +- know how to editing text files (I try to focus on generic editors like emacs, + vim, etc...), +- know the basic usage of =git= + +If you don't know that, your journey with this book might be a bit difficult but +I'll do my best to not make it impossible. + +** TO-CLEAN Opinionated + +Keep in mind that Haskell has a very active and open ecosystem. +And the language itself let you make very different choices to the fundamentals. + +This book is very opinionated, because I wanted to be efficient in learning +fast for some specific kind of personalities. + +It might not be for you. +One of my goal is to shortcircuit some classic learning detour. + +For a lot of decisions I generally make only one choice. I'll try to talk about +the other choices and it will be your duty to explore other choices after you +completed this book to decide which is the one that has your preference. + +Also note that this book was written in the past. And as I said Haskell +ecosystem evolve very fast. And some choices which are an evidence today might +be deprecated in a few months from now. + +Typically there are many different and concurrent web frameworks, db libs, etc.. + +** TO-CLEAN A Word about Haskell philosophy + +One Haskell main characteristic is that it tends to make the right/most secure +choice by default. + +A very simple example is that it is generally harder to write unsafe code than +to write safe and pure code. + +Also one of the reason I think Haskell is percieved as hard to learn by many +people is that you generally need to ingest a lot of concepts before being able +to be productive. +#+TODO: TODO TO-CLEAN TO-REVIEW | DONE +#+LANGUAGE: en +#+KEYWORDS: haskell +#+PROPERTY: header-args :output-dir HWP :mkdirp yes :tangle-mode (identity #o755) + +** TO-CLEAN Install a dev environment (about 30 minutes) + + Installing a dev environment should hopefully be the most boring part of this + book. But this is a necessary price to pay to really get why Haskell is + considered so great by people using it. + +**** TO-CLEAN Working environment + + A thing to note is the distinction between learning a language for personal + interrest for some personal project and learning with the goal to achieve a + "product" with some hard deadline. + + So for example, it can be nice to understand the language by playing inside a + REPL. We won't use that much in this book as the goal is not to really gain a + deeper knowledge but perhaps to be able to /use/ the language. + + The problem I try to solve in this book is to make you a professional /user/ of + Haskell more than a /contributor/ to Haskell. While I encourage everybody to + gain deeper understanding on the internals of Haskell this is not the primary + goal of this book. + + What I mean by professional /user/ is that you should have the following + features at your disposal: + + - DCVS + - Generated documentation + - Tests (Unit tests, Generative tests, Integration tests, etc...) + - Benchmark + - Continuous Integration + - Continuous Deployment + + Choices: + + - Raw: get GHC and cabal exectuable and work with that. Too long and manual + - Nix: this is really great because it's like a super make that can deal with + external dependencies. Certainly the best tool in the long term. + - Stack: fast to install focused on being user friendly. Has a lot of easy to + use features like: + - integration with docker that will make it easy to cross-compile, deploy. + - integration with nix + - easy to deal with many private repositories + - good professional starting templates + +**** TO-CLEAN Stack + + I recommend [[https://haskellstack.org][stack]]. But there are many different method to install Haskell. + Stack should be simple and straight to the point. + + If thing haven't changed since the book was written it could be installed with: + + #+BEGIN_SRC shell + curl -sSL https://get.haskellstack.org/ | sh + #+END_SRC + +**** TO-CLEAN git + + You should have =[[https://git-scm.com][git]]= installed. + +**** TO-CLEAN Editor + + You should check any of the supported editor here: + + https://github.com/rainbyte/haskell-ide-chart#the-chart-with-a-link-to-each-plug-in + + I personnaly use [[http://spacemacs.org][spacemacs]] with the haskell layer because it comes with battery + included. If you're not used to vim keybindings I believe it is easy to switch + to more classical editor keybindings easily. + + Even if I don't have a strong opinion on the editor you should choose. It should + at least be easy to support the Haskell tooling, like intero or ghc-mod. Because + it's one of the best part of Haskell. + + For example without any configuration I have the following features: + + - I see errors, warn and even code hints while I'm typing my code. + - very good code completion + - auto styling of my source code and be able to change the style of my entire + buffer + - be able to get the type of the expression under my cursor + - be able to add the type of a top level declaration + - be able to launch a repl easily loading the current code of the file I'm + currently editing + + And many other nice features. + + Note that in the past I had some problem with ghc-mod during upgrades while + intero was mostly a no problem story. + + It is also useful to have hoogle and hayoo, which are search engine focused on + Haskell. + +***** TO-CLEAN Spacemacs + + So if you want to choose spacemacs: + + 1. Install a recent emacs + 2. =git clone https://github.com/syl20bnr/spacemacs ~/.emacs.d= + 3. Launch emacs + 4. Edit your =~/.spacemacs= file to add to the layer list: + + #+BEGIN_SRC elisp + haskell + (auto-completion :variables + auto-completion-enable-help-tooltip t + auto-completion-enable-short-usage t) + #+END_SRC + + If you're not used to vim keybinding and it is too much to handle for you. + I think you can change the value of =dotspacemacs-editing-style= from ='vim= + to ='hybrid= or ='emacs= in the =.spacemacs= file. + + It should be good now. +**** TO-CLEAN Conclusion + + First you can congratulate yourself to have installed all the prerequiste to + have a great working development environment. + + I know it was already a lot of boring tasks to perform before being able to + write any line of code. But I promise it will be worth it. + + By starting with this template, you won't use the classic prelude. It's quite a + strong opinionated move. Because many classic function will be overwritten by + safer/more generic one. + + So be prepared that the actual learning route is jumping other classical + learning steps you can find in other learning resources. Don't worry I'll do my + best to make the jump as natural as possible. + #+TODO: TODO TO-CLEAN TO-REVIEW | DONE + #+LANGUAGE: en + #+KEYWORDS: haskell + #+PROPERTY: header-args :output-dir HWP :mkdirp yes :tangle-mode (identity #o755) + +* TODO Data Management + +You can think of Haskell and its Prelude as a Linux installation. +You have a kernel and a bunch of executable. + +So in order to manage your Unix env you need to know a few command line first. +Then, slowly with the need and experience, you'll learn more and more commands. + +So we'll do the same thing here. +We'll start with only what we need to start a project. +And You'll be introduced with new notion as they are needed. + +** Prelude/Protolude + +So Haskell prelude is old and it is very easy to use more modern prelude. +So I'll tend to do just that. + +*** Data Manipulation Functions +* TODO Effect Management + +In that part of the book, we'll use simple examples. Thus instead of going +directly to a full project structure we'll focus on the language. That file can +be treated as a single executable strict. + +For example: + +#+BEGIN_SRC haskell +#!/usr/bin/env stack +{- stack script + --resolver lts-13.13 + --install-ghc + --package protolude +-} +{-# LANGUAGE NoImplicitPrelude #-} +{-# LANGUAGE OverloadedStrings #-} +import Protolude + +main = putText "Hello World!" +#+END_SRC + +The firsts line are simply here to set the correct execution environment. +The real program starts after them. +Once =stack= will be installed (see the /Install a dev environment/ section) +if you put that content in a file named =hello.hs= then you can launch it with: + +#+BEGIN_SRC +> chmod +x hello.hs +> ./hello.hs +#+END_SRC + +The first time it is launched can take a little bit of time because it will +download all dependencies. The advantage of this form of distribution is that it +is a quasi self-contained exectuable. That's a good one for minimal examples. + +But after a short introduction we'll use full projects. + +We'll start by example first and all notion will be introduced as they appear. +If you find confident you could feel free to skip some descriptions and +explanations. + +** TODO Short Examples / Scripts +*** TO-CLEAN Guess a number +**** TO-CLEAN Print and read things + +Now let's modify the code of =main= to print things. +First comment the import line for =Lib=. +Haskell comment are =--= till the end of the line or ={- .... -}= +for multiline comments. +Without this comment you'll get a warning that this import is unused. +And by default we compile using =-Werror= flag to GHC which tell that the +compilation should fail also on warnings as well as on errors. + +The default template tend to be a professional environment and has more +restrictions in order to maximize confidence in quality. + +#+BEGIN_SRC haskell + #!/usr/bin/env stack + {- stack script + --resolver lts-11.6 + --install-ghc + --package protolude + -} + {-# LANGUAGE NoImplicitPrelude #-} + {-# LANGUAGE OverloadedStrings #-} + import Protolude + + main = putText "Hello, world!" +#+END_SRC + +Simple and natural. +Now let's ask your name. + +#+BEGIN_SRC haskell + #!/usr/bin/env stack + {- stack script + --resolver lts-11.6 + --install-ghc + --package protolude + -} + {-# LANGUAGE NoImplicitPrelude #-} + {-# LANGUAGE OverloadedStrings #-} + import Protolude + + main = do + putText "What is your name?" + name <- getLine + putText ("Hello " <> name <> "!") +#+END_SRC + +We can try that in the REPL (GHCI). You should be able to start it from your +editor. For example in spacemacs I can load the current buffer (open file) in +the REPL with =SPC m s b=. + +You could also start the repl in a terminal with =stack ghci= +And then load the module with =:l hello_name.hs=. +The =:l= is a shortcut for =:load=. + +#+BEGIN_SRC + > stack ghci + + Warning: No local targets specified, so ghci will not use any options from your package.yaml / *.cabal files. + + Potential ways to resolve this: + ,* If you want to use the package.yaml / *.cabal package in the current directory, use stack init to create a new stack.yaml. + ,* Add to the 'packages' field of ~/.stack/global-project/stack.yaml + + Configuring GHCi with the following packages: + GHCi, version 8.2.2: http://www.haskell.org/ghc/ :? for help + Loaded GHCi configuration from /private/var/folders/bp/_8thkcjd4k3g81mpxtkq44h80000gn/T/ghci70782/ghci-script + Prelude> :l hello_name.hs + [1 of 1] Compiling Main ( hello_name.hs, interpreted ) [flags changed] + Ok, one module loaded. + ,*Main> main + What is your name? + Yann + Hello Yann! +#+END_SRC + +But you should also simply run it from command line: + +#+BEGIN_SRC + > ./hello_name.sh + What is your name? + Yann + Hello Yann! +#+END_SRC + +OK simple enough. + +But let's take a moment to understand a bit more what's going on. + +We started with the =do= keyword. +It's a syntactical sugar that helps in combining multiple lines easily. +Let's take a look at the type of each part. + +#+BEGIN_SRC haskell + putText :: Text -> IO () +#+END_SRC + +It means that =putText= is a function that take a =Text= as parameter and return +an =IO ()=. +Mainly =IO ()= simply means, it will return =()= (nothing) while doing some IO +or border effect. +The border effect here being, writing the text to the standard output. + +#+BEGIN_SRC haskell + putText "What is your name?" :: IO () +#+END_SRC + +So yes this line make an IO but returns nothing significant. + +#+BEGIN_SRC haskell + name <- getLine +#+END_SRC + +The function =getLine= will read from standard input and provide the line read +and send the value as a =Text=. If you look at the type of =getLine= you have: + +#+BEGIN_SRC haskell + getLine :: IO Text +#+END_SRC + +And that means that to be able to retrieve and manipulate the Text returned by +in an "IO context" you can use the =<-= notation. +So in the code the type of =name= is =Text= + +More generally if =foo :: IO a= then when you write + +#+BEGIN_SRC haskell + do + x <- foo :: IO a +#+END_SRC + +Then the type of =x= is =a=. + +Finally the last line: + +#+BEGIN_SRC haskell + putText ("Hello " <> name <> "!") +#+END_SRC + +=putText= take a =Text= as argument so: =("Hello " <> name <> "!") :: Text=. + +So =(<>)= is the infix operator equivalent to the function =mappend=. +Here are equivalent way to write the same thing: + +#+BEGIN_SRC haskell + "Hello" <> name <> "!" + "Hello" `mappend` name `mappend` "!" + + mappend "Hello" (mappend name "!") + (<>) "Hello" ((<>) name "!") +#+END_SRC + +So in Haskell if your function contains chars it will be a prefix function. +If your function contains special chars then it is considered to be an infix +operator. + +You can use your function as infix if you put "`" around it name. +And you can make your operator prefix if you put it inside parentheses. + +So you should have remarqued a pattern here. +Which is really important. Each line of a =do= bloc has a type of =IO a=. + +#+BEGIN_SRC haskell + main = do + putText "What is your name?" :: IO () + name <- getLine :: IO Text + putText ("Hello " <> name <> "!") :: IO () +#+END_SRC + +So whenever you have an error message try to think about the type of your +expression. + +Another very important aspect to notice. +The type of ="Hello " <> name <> "!"= is =Text= not =IO Text=. +This is because this expression can be evaluated purely. +Without any side effect. + +Here we see a clear distinction between a pure part of our code and the impure +part. + +#+BEGIN_QUOTE + + +*☞ Pure vs Impure* (function vs procedure) + +That is one of the major difference between Haskell and other languages. +Haskell provide a list of function that are considered to have border effects. +Those functions are given a type of the form =IO a=. + +And the type system will restrict the way you can manipulate function with type +=IO a=. + +So, first thing that might be counter intuitive. +If an expression has a type of =IO a= it means that we potentially perform a +side effect and we "return" something of type =a=. + +And we don't want to ever perform a side effect while doing any pure evaluation. +This is why you can't write something like: + +#+BEGIN_SRC haskell + -- DOESN'T COMPILE + main = do + putText ("Hello " <> getLine <> "!") +#+END_SRC + +Because you need to "traverse" the =IO= barrier to get back the value after the +evaluation. +This is why you NEED to use the =<-= notation. +Now knowing if a code is potentially making any side effect is /explicit/. +#+END_QUOTE + +***** TO-CLEAN Strings in Haskell digression + +Generally working with string is something you do at the beginning of learning a +programming language. +It is straightforward. +In Haskell you have many different choices when dealing with Strings depending +on the context. +But let just say that 95% of the time, you'll want to use Strict =Text=. + +Here are all the possible choices: + +- =String=: Just a list of =Char= very inefficient representation, +- =Text=: UTF-16 strings can be Lazy or Strict, +- =Bytestring=: Raw stream of =Char= and also =Lazy.Bytestring=. + +That is already 5 different choices. But there is another package that provide other string choices. +In =Foundation= the strings are =UTF-8=. + +Hmmm... so much choices. + +A rule of thumbs is to never use =String= for anything serious. +Use =Text= most of the time because they support encoding. +Use =Bytestring= if you need efficient bytes arrays. + +By using Protolude, we naturally don't use =String=. + +**** TO-CLEAN Guess my age program + +So far so good. +But the logic part of the code should be in a library in =src/= directory. +Because this part is easier to test. + +The =src-exe/Main.hs= should be very minimalist, so now let's change its content +by: + +#+BEGIN_SRC haskell + #!/usr/bin/env stack + {- stack script + --resolver lts-11.6 + --install-ghc + --package protolude + -} + {-# LANGUAGE NoImplicitPrelude #-} + {-# LANGUAGE OverloadedStrings #-} + import Protolude + + guess :: IO () + guess = undefined + + main :: IO () + main = do + guess + putText "Thanks for playing!" +#+END_SRC + +We know that the type of guess must be =IO ()=. +We don't know yet what the code will be so I just used =undefined=. +This way the program will be able to typecheck. + +The next step is to define the ~guess~ function. + +#+BEGIN_SRC haskell + #!/usr/bin/env stack + {- stack script + --resolver lts-11.6 + --install-ghc + --package protolude + -} + {-# LANGUAGE NoImplicitPrelude #-} + {-# LANGUAGE OverloadedStrings #-} + import Protolude + + guess :: IO () + guess = guessBetween 0 120 + + guessBetween :: Integer -> Integer -> IO () + guessBetween minAge maxAge = do + let age = (maxAge + minAge) `div` 2 + if minAge == maxAge + then putText ("You are " <> show minAge) + else do + putText ("Are you younger than " <> show age <> "?") + answer <- getLine + case answer of + "y" -> guessBetween minAge (age - 1) + _ -> guessBetween (if age == minAge then age + 1 else age) maxAge + + main :: IO () + main = do + guess + putText "Thanks for playing!" +#+END_SRC + +So going from there we declared the =guess= function to call the =guessBetween= +function with the two paramters 0 and 120 to guess an age between 0 and 120. + +And the main function is a classic recursive function. +We ask for each age if the user is younger than some age. + +the =let= keyword permit to introduce pure values in between =IO= ones. +so =age = (maxAge + minAge) `div` 2= is mostly straightforward. +Note that we manipulate =Integer= and so that mean =`div`= is the integer division. +so =3 `div` 2 == 1=. + +We see that working in IO you can put print statements in the middle of your +code. First remark we used a recursive function. In most imperative programming +languages explicit loops are preferred to recursive functions for efficiency reasons. +That shouldn't be the case in Haskell. + +In Haskell recursive functions are the natural way to program things. + +Important Remarks to note: +- to test equality we use the =(==)= operator. +- Haskell is lazy, so the =age= value is only computed if needed. So if you are + in the case where =minAge == maxAge=, =age= value is not evaluated. +- In Haskell =if .. then .. else ..= form always have an else body. There is no + Implicit "no result" value in Haskell. Each expression need to return + something explicitely. Even if it is the empty tuple =()=. + +So now here we go: + +#+BEGIN_SRC + > stack build + > stack exec -- guess-exe + Are you younger than 60? + y + Are you younger than 29? + n + Are you younger than 44? + y + Are you younger than 36? + n + Are you younger than 39? + n + Are you younger than 41? + y + Are you younger than 39? + n + You are 40 + Bye! +#+END_SRC + +We see we can still make the program better. +For example, the same question is asked twice in that example. +Still, it works. + +*** TO-CLEAN Guess a random number + +Let's write another slightly more complex example. +Instead of guessing the age of somebody. +This will be the role of the user to guess a random number choosen by the +program. + +First we'll need to generate random numbers. +To that end we'll use a the =random= package as a new dependency. + +You can get more information either on hackage or on stackage: + +- https://hackage.haskell.org/package/random +- https://www.stackage.org/lts-11.7/package/random + +Hackage is the official place where to put Haskell public libraries. +Stackage works in conjunction with =stack= and mainly it takes care of having a +list of packages version working together. +So that means that all packages in an LTS (Long Term Support) release can work +together without any build conflict. + +Now let's use that package. +Notice the added =--package random= argument. + +We'll start by writing a =guessNumber= function: + +#+BEGIN_SRC haskell + #!/usr/bin/env stack + {- stack script + --resolver lts-11.6 + --install-ghc + --package protolude + --package random + -} + {-# LANGUAGE NoImplicitPrelude #-} + {-# LANGUAGE OverloadedStrings #-} + + import Protolude + + import System.Random (randomRIO) + + ... + + -- | Choose a random number and ask the user to find it. + guessNumber :: IO () + guessNumber = do + n <- randomRIO (0,100) + putText "I've choosen a number bettween 0 and 100" + putText "Can you guess which number it was?" + guessNum 0 n + + -- | Given a number of try the user already made and the number to find + -- ask the user to find it. + guessNum :: Int -> Int -> IO () + guessNum nbTry nbToFound = undefined +#+END_SRC + +So for now we just focus on how to get a random number: + +#+BEGIN_SRC haskell + do + n <- randomRIO (0::Int,100) + -- do stuff with n +#+END_SRC + +You NEED to use the =<-= notation inside a =do= bloc. +If you try to use =let n = randomRIO (0,100)= it will fail because the +types won't match. + +And that's it! + +Now to write the =guessNum= function, we'll write a classical recursive function: + +#+BEGIN_SRC haskell + -- | Given a number of try the user already made and the number to find + -- ask the user to find it. + guessNum :: Int -> Int -> IO () + guessNum nbTry nbToFound = do + putText "What is your guess?" + answer <- getLine + let guessedNumber = readMaybe (toS answer) + case guessedNumber of + Nothing -> putText "Please enter a number" + Just n -> + if n == nbToFound + then putText ("You found it in " <> show (nbTry + 1) <> " tries.") + else do + if n < nbToFound + then putText "Your answer is too low, try a higher number" + else putText "Your answer is too high, try a lower number" + guessNum (nbTry + 1) nbToFound +#+END_SRC + +Let's read the program line by line: + +- ~putText "What is your guess?"~ should be straightforward. +- ~answer <- getLine~ So the ~getLine~ read from standard input and returns the + line entered by the user. The line will be put in the ~answer~ variable. +- ~let guessedNumber = readMaybe (toS answer)~: there are a few things to tell about this line. + +If you open GHCI and ask the type for each interresting symbol here is what you get: + +#+BEGIN_SRC + λ :t getLine + getLine :: IO Text + + λ :t toS + toS :: StringConv a b => a -> b + + λ :t readMaybe + readMaybe :: Read a => GHC.Base.String -> Maybe a +#+END_SRC + +- ~answer~ comes from ~getLine :: IO Text~ so ~answer~ should have the type ~Text~. +- Now we want to read this ~Text~ and see if this is a number and compare it to another ~Int~. +- To transform the number we don't use a function ~textToInt~ we simply use a + quite generic function ~readMaybe~ that take some ~String~ and try to + transform that to some type. For our specific case, the compiler is able to + figure out the type we want to transform the text into is ~Int~. Take the + time to digest that: ~Int~ is specified in the type signature of the + ~guessNum~ function so the compiler could discover that ~readMaybe~ should + return a ~Maybe Int~. How does he do that? Let's follow: + 1. see a ~n == nbToFound~ so we can deduce ~n~ and ~nbToFound~ have the same type. + 2. Reading the type signature of the function it is clear ~nbToFound~ is of + type ~Int~ (it's the second argument of a function with type ~Int -> Int -> IO ()~) + 3. Then ~n~ is generated from a pattern matching; the case ~Just n~ which + could be the the result of the ~readMaybe~ function. So we can deduce + that the ~a~ in the type signature of ~readMaybe~ is ~Int~ for this specific case. +- so ~guessedNumber :: Maybe Int~, if the user enter something that cannot be + transformed in number from a string then ~guessedNumber~ would be equal + to ~Nothing~ and we ask the user to enter a number. If the user entered a + number the type will be ~Just n~ were ~n~ will be an ~Int~. +- We compare the ~guessedNumber~ to the number to found ~nbToFound~. +- If the user found the right number we stop here by displaying the number of try. +- If the user hasn't found the number, depending on its value we tell the user + it's either too low or too high and we call the same function, this time, we + increment the number of try. + +The full program is then: + +#+BEGIN_SRC haskell + #!/usr/bin/env stack + {- stack script + --resolver lts-11.6 + --install-ghc + --package protolude + --package random + -} + {-# LANGUAGE NoImplicitPrelude #-} + {-# LANGUAGE OverloadedStrings #-} + + import Protolude + + import System.Random (randomRIO) + + main :: IO () + main = guessNumber + + -- | Choose a random number and ask the user to find it. + guessNumber :: IO () + guessNumber = do + n <- randomRIO (0,100) + putText "I've choosen a number bettween 0 and 100" + putText "Can you guess which number it was?" + guessNum 0 n + + -- | Given a number of try the user already made and the number to find + -- ask the user to find it. + guessNum :: Int -> Int -> IO () + guessNum nbTry nbToFound = do + putText "What is your guess?" + answer <- getLine + let guessedNumber = readMaybe (toS answer) + case guessedNumber of + Nothing -> putText "Please enter a number" + Just n -> + if n == nbToFound + then putText ("You found it in " <> show (nbTry + 1) <> " tries.") + else do + if n < nbToFound + then putText "Your answer is too low, try a higher number" + else putText "Your answer is too high, try a lower number" + guessNum (nbTry + 1) nbToFound +#+END_SRC + +which once executed: + +#+BEGIN_SRC + > ./guess_number.hs + I've choosen a number bettween 0 and 100 + Can you guess which number it was? + What is your guess? + 50 + Your answer is too low, try a higher number + What is your guess? + 75 + Your answer is too low, try a higher number + What is your guess? + 90 + Your answer is too high, try a lower number + What is your guess? + 83 + Your answer is too low, try a higher number + What is your guess? + 87 + You found it in 5 tries. +#+END_SRC + +**** TO-CLEAN What did we learn so far? + +So up until now, if you followed. You should be able to "reproduce" and make +minimal changes. +But I am certain than it still be difficult to make some changes. +It is time to learn some general principles. +I know it might be a bit repetitive but its important to be certain to ingest +those informations. + +A generic function of type ~IO ()~ typically =main= should look like: + +#+BEGIN_SRC haskell + f :: IO a + f = do + α <- f1 + β <- f2 + γ <- f3 + δ <- f4 + f5 +#+END_SRC + +where each expression =fi= is of type =IO a= for some =a=. +You can use any value =α=, =β=, etc‥ as a parameter. +In order to be valid. +The last expression must have the same type as =f=. +so here =f5 :: IO a=. + +Now if I give you the following functions: + +- ~getLine :: IO Text~ that read a line from stdin. +- ~putText :: Text -> IO ()~ that read a line from stdin. + +With that you have the ability to read stdin and print things. + +- ~if τ then f1 else f2~ where =τ :: Bool= and the type of ~f1~ and ~f2~ must be the + same. Generally this is denoted by: =:type f1 ~ :type f2= and that type + will be the same as the entire ~if ‥ then ‥ else ‥~ expression. +- you can compare things that can be compared with ~<~, ~<=~, ~>~, ~>=~, ~==~, ~/=~ (different). +- you can concatenate things that could be concatenated (like Text) with ~<>~ +- you can transform things as Text with ~show~ in particular numbers. + +So that is a few number of component but they are all composable. +And so far we only needed that to write our first programs. + +Haskell libs will provide you with a lot more base functions but also a lot more +composition functions. + +*** TODO Command Line Application + +Another thing you might want to achieve at first is to retrieve arguments for a +command line application. + +**** TO-CLEAN Basic + +The simplest way to retrieve parameters to a command line is to use the ~getArgs~ function. + +#+BEGIN_SRC haskell + getArgs :: IO [String] +#+END_SRC + +Here is a minimal example. + +#+BEGIN_SRC haskell + #!/usr/bin/env stack + -- stack --resolver lts-11.6 script + {-# LANGUAGE OverloadedStrings #-} + {-# LANGUAGE NoImplicitPrelude #-} + import Protolude + import System.Environment (getArgs) + + main :: IO () + main = do + arguments <- getArgs + case head arguments of + Just filename -> die ("The first argument is: " <> toS filename) + Nothing -> die "Please enter a filename" +#+END_SRC + +#+BEGIN_SRC + > ./cmdline-basic.sh foo + The first argument is: foo + > ./cmdline-basic.sh + Please enter a filename +#+END_SRC + +If you have a very basic command line it could be good enough. +But if you plan to have more things to configure you can consider +to use a library to parse options. + +**** TODO Option Parsing + +For that we will use the =optparse-generic= package. + + +#+BEGIN_SRC haskell + #!/usr/bin/env stack + {- stack script + --resolver lts-11.6 + --install-ghc + --package protolude + --package optparse-generic + -} + {-# LANGUAGE NoImplicitPrelude #-} + {-# LANGUAGE OverloadedStrings #-} + import Protolude + import System.Environment (getArgs) + + main :: IO () + main = do + arguments <- getArgs + case head arguments of + Just filename -> die ("The first argument is: " <> toS filename) + Nothing -> die "Please enter a filename" +#+END_SRC + +*** TODO File Access +*** TODO Daemons & Logging +** TODO Intermediate +*** TO-CLEAN Stack template + +☞ As a first projet a lot of new concept will be introduced. Don't be +discouraged by that. + +Let's create a project with a sane and modern file organisation. + +I made a stack templates largely inspired by =tasty-travis= template. It will +provide a bootstrap for organizing your application with tests, benchmarks and +continuous integration. + +This template provide a file organisation for your projects. + +Mainly do jump into programmin you could theoretically just download the binary +of the main Haskell compiler GHC to your compiler and compile each file with +=ghc myfile.hs=. But let's face it. It's not suitable for real project which +need more informations about it. + +So let's start with a sane professional organisation for your files. + #+BEGIN_COMMENT -/THIS IS A WORK IN PROGRESS/ - -*CONTRIBUTORS* - -- To extract all source code files do ~org-babel-tangle~ (, b t) in spacemacs -- To go to an imported file use (, ') - -Audience of this book: - -- people that are asked to work in Haskell but don't know much about it -- people that already tried a bit of Haskell and want to go further -- people that need to make a professional level product with Haskell -- people that are ok not understanding every detail yet but need the job to be - done - -Book is not for: - -- Total beginner that just want to know what Haskell is about -- Haskell contributors that already know a lot of things about Haskell - - -Goals: - -After finishing this book an reader should be able to: - -- script in Haskell instead of bash -- write a REST API with a swagger documentation and deploy it -- perhaps write a frontend app (I don't know looks like a LOT of work) - +****** TODO modify the URL to use a better URL: torrent / IPFS #+END_COMMENT -#+Include: HWP/1_Introduction.org -#+Include: HWP/2_Install.org -#+Include: HWP/3_Intermediate.org -#+Include: HWP/4_Production.org -#+Include: HWP/5_Advanced.org -#+Include: HWP/6_Appendices.org +#+BEGIN_SRC shell + stack new guess https://git.io/vbpej +#+END_SRC + +After that, this should generate a new ~guess~ directory with the following +files: + +#+BEGIN_SRC + > tree + . + ├── CHANGELOG.md + ├── LICENSE + ├── README.md + ├── Setup.hs + ├── guess.cabal + ├── package.yaml + ├── src + │ └── Lib.hs + ├── src-benchmark + │ └── Main.hs + ├── src-doctest + │ └── Main.hs + ├── src-exe + │ └── Main.hs + ├── src-test + │ └── Main.hs + ├── stack.yaml + └── tutorial.md + + 5 directories, 13 files +#+END_SRC + +Most of your source code should be in the =src= directory. Generally =src-exe= +should be a minimal code that could handle the =main= function to start your +application. We'll talk about other parts later in the book but most other file +should be quite straightforward. + +Edit the file =src-exe/Main.hs= + +The file contains: + +#+BEGIN_SRC haskell + import Protolude + + import Lib (inc) + + main :: IO () + main = print (inc 41) +#+END_SRC + +To compile it do a + +#+BEGIN_SRC + > stack build + > stack exec -- guess-exe + 42 +#+END_SRC + +So that program print 42 and stop. + +*** TODO DB Access +**** NoSQL (Redis looks easy) +**** Stream DB (Kafka or NATS, etc...) +**** SQL (SQLite & Postgres) +Not sure about that part. Perhaps this should move in the Production section +*** TODO REST API +**** TODO Servant +**** TODO JSON manipulation +**** TODO Swagger-UI +** TODO Intermediate Conclusion + +Congratulation for going this far. Now you should be able to work in Haskell at +least as well as in any other programming language. + +Now there are different directions: + +- learning more libraries +- learn to optimise code to make it as fast as C +- learn to understand details of the compilation and Haskell +- learn tips and tricks +- learn more about abstractions and type classes +- learn parallel and concurrent programming +- learn to deploy like a pro using nix + +The order in which to learn all thoses things can be very different for everty need. +#+TODO: TODO TO-CLEAN TO-REVIEW | DONE +#+LANGUAGE: en +#+KEYWORDS: haskell +#+PROPERTY: header-args :output-dir HWP :mkdirp yes :tangle-mode (identity #o755) + +* TODO Make Production Quality Products +** TODO Dhall /Maybe/ +** TODO Error Handling + +** TODO Unit testing / doctests +** TODO Generative Testing +** TODO Enhance reproductibility with docker +** TODO Enhance reproductibility with nix +** TODO How to deploy? + +There are plenty of ways de deploy + +*** TODO Trashy and easy + +Compile in docker and copy the binary. + +*** TODO With =nix= and =nixops= +#+TODO: TODO TO-CLEAN TO-REVIEW | DONE +#+LANGUAGE: en +#+KEYWORDS: haskell +#+PROPERTY: header-args :output-dir HWP :mkdirp yes :tangle-mode (identity #o755) + +* TODO Enhance Code Quality +** TODO Type tricks + + +*** TODO New Types +*** TODO Phantom Types +** TODO Useful Abstractions +*** TODO Monoid +*** TODO Functors +*** TODO Applicative +*** TODO Monads +*** TODO Arrows +*** TODO Foldable +*** TODO Traversable +** TODO Useful Abstractions for Applications +*** TODO No organisation, everything in IO +*** TODO Handler Pattern + +See the talk from Jasper Van Der Jeught + +In my opinion as efficient as MTL, Free, Effects, but with more verbosity and +repetitions. + +*** TODO Custom Monad +*** TODO MTL +*** TODO Free +*** TODO Effects +** TODO Lenses + +This will only be an introduction for being an user of the library. + +** TODO Generics and lens-generic +** TODO Code organisation +*** TODO No organisation, everything in IO +*** TODO Custom Monad +*** TODO Handler Pattern +*** TODO MTL +*** TODO Free +*** TODO Effects +** TODO Recognize some classical abstractions +*** TODO Algebra +*** TODO Catamorphisms +*** TODO Free & Interpreters +#+TODO: TODO TO-CLEAN TO-REVIEW | DONE +#+LANGUAGE: en +#+KEYWORDS: haskell +#+PROPERTY: header-args :output-dir HWP :mkdirp yes :tangle-mode (identity #o755) + +* TODO Appendices + +** TODO Papers You Should have read diff --git a/TODO.org b/TODO.org index 6e648860..68b25e7e 100644 --- a/TODO.org +++ b/TODO.org @@ -1,6 +1,6 @@ #+Title:TODO #+Author: Yann Esposito -#+TODO: TODO IN-PROGRESS WAIT | DONE CANCELED +#+TODO: TODO IN-PROGRESS HOLD WAITING | DONE CANCELED #+COLUMNS: %TODO %3PRIORITY %40ITEM(Task) %17Effort(Estimated Effort){:} %CLOCKSUM %8TAGS(TAG) Paris 7h10 - 8h45, à Londre, 15 mars @@ -29,7 +29,6 @@ Les languages de programmation fonctionnels ont introduits récemment l'usage de structures de données immuables. Nous verrons comment celà facilite énormément l'écriture de programmes parallèles et concurrents. - *** DONE Virement maman 3000€ CLOSED: [2019-03-29 Fri 09:23] - IBAN: FR28 3000 2028 2100 0005 8155 N89 @@ -58,8 +57,8 @@ l'écriture de programmes parallèles et concurrents. Tel: 09 80 89 95 31 ** Meetup Lamda Riviera :geek: -** Projects :dev:geek: -*** TODO Write a comments system +** Projects :dev:geek: +*** IN-PROGRESS Write a comments system Requirements: @@ -109,7 +108,8 @@ Requirements: https://man.sr.ht/installation.md) - Something rawer like gpm? **** TODO Enhance Domain Name hoster to better support letsencrypt (typically CAA I think) -*** =gpm= todo / wiki / docs, etc... :dweb: +*** CANCELED =gpm= todo / wiki / docs, etc... :dweb: + CLOSED: [2019-04-14 Sun 20:00] **** DONE Doc CLOSED: [2018-11-17 Sat 13:07] Write a tool to handle the following workflow. @@ -165,8 +165,8 @@ Requirements: CLOSED: [2019-03-03 Sun 15:08] ***** DONE Ignore patterns CLOSED: [2019-03-02 Sat 20:07] -***** TODO [#C] Use a split and Index on n-grams to match for all bots fast -***** TODO [#C] User regex? +***** HOLD [#C] Use a split and Index on n-grams to match for all bots fast +***** HOLD [#C] User regex? **** TODO Enhance Producer by retrieving all post/comments [[https://intoli.com/blog/f5bot/][f5bot]] **** TODO Send mails on matches **** TODO Dev/Ops Improvement diff --git a/agenda.org b/agenda.org new file mode 100644 index 00000000..638140fa --- /dev/null +++ b/agenda.org @@ -0,0 +1,13 @@ +* Agenda +** TODO yet another thing to test + :LOGBOOK: + CLOCK: [2019-04-15 Mon 00:35]--[2019-04-15 Mon 00:35] => 0:00 + :END: + [2019-04-15 Mon 00:35] + [[file:~/.spacemacs::'(org-agenda-files%20'("~/.deft/agenda.org"))]] +** TODO xxx + :LOGBOOK: + CLOCK: [2019-04-15 Mon 00:36]--[2019-04-15 Mon 00:36] => 0:00 + :END: + [2019-04-15 Mon 00:36] + [[file:~/.spacemacs::'(org-refile-targets%20'((nil%20:maxlevel%20.%209)]] diff --git a/cisco-epic-feature-flag-by-env.org b/cisco-epic-feature-flag-by-env.org new file mode 100644 index 00000000..e00a5e85 --- /dev/null +++ b/cisco-epic-feature-flag-by-env.org @@ -0,0 +1,72 @@ +#+Title:Cisco Epic Feature Flag by Env +#+Author: Yann Esposito +#+LANGUAGE: en +#+TODO: TODO IN-PROGRESS WAIT | DONE CANCELED + +* Feature Flag by Env :TOC_3_gh:QUOTE: +#+BEGIN_QUOTE + - [[#requirement][Requirement]] + - [[#current-status][Current status]] + - [[#proposed-solution][Proposed Solution.]] + - [[#feature-flag-block-in-configedn][Feature Flag Block in =config.edn=]] +#+END_QUOTE + +** Requirement + +> Craig Brozefsky Yesterday, 17:04 +> Matt: No orgs in EU should have devices scope + +*** Current status + +Until now we managed feature flag for an entire release. As such the +feature-flag was 100% held in the code, not in the deployement conf. + +There are two methods used until today to manage feature-flags: + +1. scopes +2. service launch (in bootstrap.cfg) +3. control via config (in config.edn) + +The current requirement only talks about the =sse= scope. +In fact we could (should) also prevent the =sse-service= to be launched. + +Note that scope handling is generally not trivial: + +- CTR use the scope as the single dimension to handle authorizations. One + consequence is that the notion of /role/ is not really meaningful in the CTR + code. The /role/ is only used from the info provided by the IdP and then + interpreted as a set of scopes (which can change dynamically, for exemple we + can attribute additional scopes to some org or user). It is also planned to + provide the ability for admin users to change the scopes of other users of + their org. + +** Proposed Solution. + +*** Feature Flag Block in =config.edn= + +That way it would be possible to not only handle scopes but also manage the +feature flag in some specific part of the code. Typically we could use that flag +to ignore some conf, and to not initialize fully some service. + +**** Service Launch Handling + +- If a service is started but its presence is not necessary when the feature + flag is off. The service should not really init itself fully and only return a + nil context and the methods should also returns nil silently + +**** Scope Handling + +Depending of the feature flag we might add an additional step during login we +might "add" some new scopes and "remove some". If that's the case we might also +change the JWT version dyamically. + +I would suggest something like: + + +#+BEGIN_SRC clojure +(defn dyn-jwt-version [activated-features] + (string/join "-" (cons static-jwt-version activated-features)) +#+END_SRC + +That would produce version such as: =v1.23=, =v1.23-sse=, =v1.23-sse-scim=, +etc... diff --git a/journal.org.gpg b/journal.org.gpg index 04a26dc6..6d7edb5c 100644 Binary files a/journal.org.gpg and b/journal.org.gpg differ diff --git a/refile.org b/refile.org new file mode 100644 index 00000000..6c39ee17 --- /dev/null +++ b/refile.org @@ -0,0 +1,4 @@ +#+FILETAGS: REFILE +* Tasks +* Notes +* Agenda