Skip to content

Latest commit

 

History

History
4311 lines (2769 loc) · 115 KB

dev-log.md

File metadata and controls

4311 lines (2769 loc) · 115 KB
Raw

August 2011 - first days at Revelytix

MONDAY

Install ubuntu 10.10 full disk installation

alt-F2 launches application make Caps Lock key another ctrl key System->Preferences->Keyboard Layouts Options "Ctrl Key position"

sudo apt-get install
git
hg
emacs
maven2
openjdk-6-jdk
dropbox*
google-chrome-stable*
tree
vim

GitHub create new RSA public key, add to GitHub account https://help.github.com/articles/generating-ssh-keys

skype

git clone clojure; build from src

need to add repo

Install Leiningen (install script from github page into ~/bin/, chmod +x, add to $PATH, "lein self-install") lein deps lein compileDeps lein compile lein test lein upgrade

Install ~/.hgrc

configure .m2 - copy settings.xml to ~/m2

emacs

M-x package-install then clojure-mode

hg clone http://www.assembla.com/revelytix-common

knoodel cert https://build.knoodel.net/maven-repo bottom-right corner->details->export .net file, then

sudo keytool -import -alias root -file /home/scott/Downloads/gd_intermediate.crt -storepass changeit -trustcacerts -keystore /usr/lib/jvm/java-6-openjdk/jre/lib/security/cacerts
sudo keytool -import -alias cross -file /home/scott/Downloads/gd_cross_intermediate.crt -storepass changeit -trustcacerts -keystore /usr/lib/jvm/java-6-openjdk/jre/lib/security/cacerts
sudo updatedb
locate foo
locate bar

TUESDAY

Leiningen lein new project-name lein deps lein compileDeps lein compile lein dist lein swank

Emacs M-x slime-connect C-c C-k load/compile C-c M-p repl at namespace C-x C-e eval expr C-c v eval buffer M-. func src M-or C-x b ???

WEDNESDAY

Clojure

  • replace if (do ( with when( unless there's an else
  • doseq drop-in non-lazy replacement for for

Emacs

  • paredit cheat sheet
  • M-! shell command "tree"* tw There should be a Clojure Lint called Clint
  • in repl use (in-ns 'somenamespace)
  • vc-mode C-x V D
  • M-/ hippie complete
  • M-x rgrep
    • P N forward/back navigation of results
  • emacs kill ring
    • M-y cycle through kill ring for yank
    • M-x browse-kill-ring (need to install module)

THURSDAY

Ubuntu crash, and recover w/ 11.04 install disk (and help from Ryan)

  • reinstall gnome-do, chrome, git, hg, emacs, java, dropbox, etc.

Linux tips

  • Ctrl-Alt-F2 - bring up terminal [Ctrl-Alt-F1..F8 switch between terminals]

  • restart Gnome display manager:

      sudo /etc/init.d/gdm restart - restart Gnome display manager

FRIDAY

install skype https://help.ubuntu.com/community/Skype enable Canonical partners repo (use Software Center)

sudo apt-get install skype

wiki https://revelytix.servehttp.com:553/wiki/Human_Resources

sent website feedback to Paul

installed Federator, ran demo

emacs color-theme debugging (thanks Ryan) http://blog.dskang.com/2011/05/14/fixing-the-color-theme-problem-with-the-emacs-starter-kit/

correct Federator README

MONDAY 8/29/11

keytool -list -keystore /usr/lib/jvm/java-6-openjdk/jre/lib/security/cacerts -alias root -v

sudo apt-get install tk8.5 (for wish)

Mercurial commit: hg pull -u or hg in hg view hg heads hg rebase -b 2192 -d tip hg outgoing hg push

TUESDAY 8/30/11

backup sudo apt-get install backintime-common sudo apt-get install backintime-gnome sudo apt-get install cryptsetup

WEDNESDAY 8/31/11

install mysql thru synaptic

TUESDAY 9/6/11

hg glog -l 5

C-A-Backspace - ubuntu restart session (recover from crash)

FRIDAY 9/9/11

Build https://wiki.revelytix.com:553/wiki/BuildServer

ssh -p 3122 -i <build server key> -o "ServerAliveInterval=10" [email protected]
screen [-r]
jenkins@build:~/jobs/Spyder/workspace$ /usr/local/revelytix_data/lein test revelytix.spyder.tool.test-tool

Wed 9/28

(println "<!><!><!><!> ttl")
(clojure.stacktrace/print-stack-trace (Exception.))


(comment (def tuple {
                 :b (new-literal*
                     {:lexical "2001-01-01 01:01:01.0",
                      :datatype (java.net.URI. "http://www.w3.org/2001/XMLSchema#dateTimeStamp")}),
                 :name (new-literal* {
                                      :lexical "bill",
                                      :datatype (java.net.URI. "http://www.w3.org/2001/XMLSchema#string")})})

Monday 10/24/11

parser stuff (in repl)

revelytix.grammar.sql> 
(revelytix.parsergen.slr/output-canonical-states)
(dbg-parse "sELeCt a from b")
(apply dbg-parse *1)
all-productions

Tuesday 11/1/11

EC2 ops repl session:

; SLIME 20100404
user> 
revelytix.ops> (init)
#<AmazonEC2Client com.amazonaws.services.ec2.AmazonEC2Client@1652d654>
revelytix.ops> aws
#<AmazonEC2Client com.amazonaws.services.ec2.AmazonEC2Client@1652d654>
revelytix.ops> (describe-instances (memfn getImageId) #(.. % getState getName))
(["ami-4b4ba522" "stopped"] ["ami-5b799232" "stopped"] ["ami-56936b3f" "stopped"])
revelytix.ops> (describe-images "ami-b62ac0df" "ami-4b4ba522" "ami-56936b3f" "ami-5b799232" [:imageId :name :description])
({:description "Oracle on Ubuntu 10.04 Server 64-bit", :name "Oracle 112_P2 Ubuntu", :imageId "ami-5b799232"} {:description "Even better as now the startup script is executable. I'm a dumbass.", :name "Oracle 112_P2 WebLogic 40G Latest Aug24-2", :imageId "ami-b62ac0df"})
revelytix.ops> (clojure.pprint/pprint (describe-images "ami-b62ac0df" "ami-4b4ba522" "ami-56936b3f" "ami-5b799232" [:imageId :name :description]))
({:description "Oracle on Ubuntu 10.04 Server 64-bit",
  :name "Oracle 112_P2 Ubuntu",
  :imageId "ami-5b799232"}
 {:description
  "Even better as now the startup script is executable. I'm a dumbass.",
  :name "Oracle 112_P2 WebLogic 40G Latest Aug24-2",
  :imageId "ami-b62ac0df"})
nil
revelytix.ops> (clojure.pprint/pprint (describe-images "ami-b62ac0df" "ami-4b4ba522" "ami-56936b3f" "blahblahblah" "ami-5b799232" [:imageId :name :description]))
; Evaluation aborted.
revelytix.ops> (clojure.pprint/pprint (describe-images "ami-b62ac0df" "ami-4b4ba522" [imageId :name :description]))
; Evaluation aborted.
revelytix.ops> (clojure.pprint/pprint (describe-images "ami-b62ac0df" "ami-4b4ba522" [:imageId :name :description]))
({:description
  "Even better as now the startup script is executable.",
  :name "Oracle 112_P2 WebLogic 40G Latest Aug24-2",
  :imageId "ami-b62ac0df"})
nil
revelytix.ops> (clojure.pprint/pprint (describe-images "ami-4b4ba522" [:imageId :name :description]))
()
nil
revelytix.ops> (map (memfn getPublicDnsName) (running-instances))
("ec2-174-129-187-23.compute-1.amazonaws.com")
revelytix.ops> (terminate-running)
#<TerminateInstancesResult {TerminatingInstances: [{InstanceId: i-cb02dea8, CurrentState: {Code: 32, Name: shutting-down, }, PreviousState: {Code: 16, Name: running, }, }], }>
revelytix.ops>

Wed 11/2/11

acpi - check battery

Thu 12/1/11

ssh tunnelling for remote Amazon EC2 mysql box

ssh -L 3360:localhost:3306 [email protected]

(see ~/bin/pf script for example)

xmonad

install xmonad (sudo apt-get install xmonad) Create /usr/share/xsessions/xmonad-gnome-session.desktop Create /usr/share/gnome-session/sessions/xmonad.session (maybe) Create /usr/share/applications/xmonad.desktop file

Mon 4/23/12

sudo rfkill list all

export JDK_HOME=/home/scott/sandbox/sunjdk-7/jdk1.7.0

trayer --edge top --align right --SetDockType true --SetPartialStrut false --expand true --width 5 --transparent true --tint 0x0000000 --height 18 --alpha 0 &

apt-get
apt-cache

M-: (setq kill-ring nil)

Dropbox git remote steps: http://stackoverflow.com/questions/1960799/using-gitdropbox-together-effectively

cd ~/Dropbox/git
git init --bare foo.git
cd ~/foo
git remote add origin ~/Dropbox/git/foo.git
git push -u origin master

Tue 1/22/2013

upgrading to emacs 24, building from source (clone from github, cd into directory, then)

sudo apt-get install autoconf automake
./autogen.sh
sudo apt-get install texinfo libgtk3-dev libgif-dev libjpeg-dev
libtiff-dev libxpm-dev libtinfo-dev
./configure --with-x-toolkit=gtk
make
src/emacs -Q    <-- test
sudo make install

terminal only (amazon aws microinstance):

sudo apt-get install texinfo libtinfo-dev ncurses-dev
./configure --without-x

Wed 1/23/2013

Use different JDK: edit ~/.bashrc, set $JDK_HOME e.g.

export JDK_HOME=/home/scott/sandbox/sunjdk-7/jdk1.7.0
export JDK_HOME=/usr/lib/jvm/java-7-openjdk

set java in path

ln -s /usr/lib/jvm/java-7-openjdk/jre/bin/java ~/bin/java

relaunch shell

bash

Thu 1/24/2013

truncate long lines in a tail

tail spyder.log | cut -b 1-80

Fri 1/25/2013

grep -r -B 1 -A 80 something.something logs/spyder.log

tar -czf foo.tar.gz bar/

Fri 2/8/2013

Emacs M-: eval M-! shell command

Fri 2/12/13 grep tricks

  • lines before/after a match

      grep -r -B 1 -A 80 something.something logs/spyder.log

  • pattern is fixed string, match only whole line

      grep -Fx "a whole line" file.log

  • remove dupes

      grep yadda yadda yadda | sort | uniq

  • truncate lines

      grep yadda yadda yadda | cut -b 1-10

append to file

>>

2/13/13

Sys admin tools: top, htop, w, iostat, nmap, du

2/19/13

Install ssh daemon

apt-get openssh-server

Restart

sudo service ssh restart

ssh with X tunnelling

ssh -X scott@your-dell-laptop

Find hostname

ifconfig
`hostname`.local

emacs C-O all grep occurences

2/22/13

gnome-terminal profiles

~/.gconf/apps/gnome-terminal/profiles/Solarized/

terminfo

/usr/share/terminfo/
/lib/terminfo/

emacs terminal stuff

~/sandbox/emacs/lisp/term/

emacs color cells

(display-color-cells (selected-frame))

or

(tty-display-color-cells (selected-frame))
C-x C-e eval in scratch buffer

colors

M-x list-colors-display

terminfo? ncurses? tput

3/1/13

create .tar.gz (gzipped) file of director(ies):

tar -czvf archive.tar.gz foo/ bar/

extract tar

tar -xzvf archive.tar.gz

3/7/13

port forwarding script:

#!/bin/sh

HOST=ec2-107-21-140-138.compute-1.amazonaws.com
exec ssh -i ~/dev/revelytix-spyder/spyder-testing.pem -o ServerAliveInterval=10 ubuntu@$HOST -L 3360:localhost:3306

update ubuntu JDK:

sudo update-alternatives --config java ;; don't do this, do below instead

http://askubuntu.com/questions/141791/is-there-a-way-to-update-all-java-related-alternatives

update-alternatives --get-selections | grep java
/usr/sbin/update-java-alternatives -l
sudo apt-get install icedtea-7-plugin
sudo /usr/sbin/update-java-alternatives -s java-1.7.0-openjdk-amd64

3/13/13

Emacs global search and replace http://emacswiki.org/emacs/DiredSearchAndReplace

C-u M-x
Dired listing switches: -lR

in dired mode, mark directories:

* /

then toggle marks to mark only files

t

(or, mark files according to regexp % m) then do search replace regexp

Q

accept all

Y

In IBuffer (C-x C-b), mark all unsaved

* u

then save all marked

S

regexp search and replace "Query" with "Transform"

M-x replace-regexp (also works in above dired steps)
\([^a-zA-Z0-9]+\)Query+
Ret
\1Transform
Ret

3/25/13

start Loom transactor:

~/dev/revelytix-bigdata/fabric/bin/transactor ~/dev/revelytix-bigdata/fabric/conf/transactor.properties > /dev/null &

7/29/13

getting Deech up-and-running

9/22/13

sudo apt-get autoremove

http://artofsimplicity.co.uk/install-dropbox-on-a-headless-ubuntu-server/

~/bin/dropbox.py help

https://www.dropbox.com/help/175/en

scp .hgrc then

hg clone https://hg.assembla.com/revelytix-bigdata/
sudo apt-get install openjdk-7-jdk (hopefully has godaddy certs, for maven repo on build server - didn't work)

(needed to do import 2 godaddy cacerts steps from build server wiki page)

1/29/14

Setup Docker:

http://docs.docker.io/en/latest/installation/ubuntulinux/

sudo sh -c "echo deb http://get.docker.io/ubuntu docker main\
> /etc/apt/sources.list.d/docker.list"

sudo apt-get install lxc-docker

Run hello world

sudo docker run -i -t ubuntu /bin/bash

5/9/14

count lines of code LOC

scott@scott-latitude:~/dev/revelytix-bigdata/fabric/src/main/resources/public/js/rvtx$ find . -name "*.html" -o -name "*.js" | xargs wc -l

6/6/14 tag loom release

hg tag -m "tag Loom 2.2.0 release (re #3557)" 2.2.0

6/9/14 update java cacerts - relevant directories

~/bin/java
/usr/lib/jvm/
/etc/ssl/certs/java/cacerts

see

https://www.assembla.com/spaces/revelytix-common/wiki/Build_Server#3.3.1_go_daddy_certificates

tried it:

~$ keytool -import -alias godaddy_intermediate_g2 -file ~/Downloads/gdig2.crt -trustcacerts -keystore /etc/ssl/certs/java/cacerts
Enter keystore password:

password

<%=revelytix-jdk-certs%>

7/7/14 Wordpress

'home' and 'site' urls can be stored either in wp-config.php (WP_HOME, WP_SITEURL) or in mysql kwcoc_options table with option_name column of home and siteurl.

Assembla->Git Migration

git --version
hg paths
mv .hgignore .gitignore

instructions https://www.assembla.com/code/revelytix-bigdata/git/repo/instructions

cloned 'fast-export', did the export

git clone git://repo.or.cz/fast-export.git

cloned fresh copy of hg repo

mkdir revelytix-bigdata-prime
hg clone https://hg.assembla.com/revelytix-bigdata revelytix-bigdata-prime/

made new empty repo

mkdir revelytix-bigdata-git
cd revelytix-bigdata-git
git init

did export

../fast-export/hg-fast-export.sh -r ../revelytix-bigdata-prime/

generate and upload ssh key to assembla profile

added remote to git repo

git remote add assembla [email protected]:revelytix-bigdata.git

push

git push --all assembla

Watch all three Godfather films

Use 'https'

https://git.assembla.com/revelytix-bigdata.git

(instead of 'git')

[email protected]:revelytix-bigdata.git

7/9/14

list processes by user

ps -ef

7/12/14 xargs example

find . -name "*.bak" -print0 | xargs -0 -I file mv file ~/old.files

7/14/14 git config

git config --global -l
git config --local -l

git config --local user.email "<my work email address>"

7/16/14

backup Jenkins, Nexus

/usr/local/revelytix_data/hudson-backups
/usr/local/revelytix_data/sonatype-work/nexus

tar/gzip director(ies)

tar czvf foo.tar.gz folder1 folder2... folderN

8/5/14

change Grub default

fgrep menuentry /boot/grub/grub.cfg
sudo zile /etc/default/grub 
sudo update-grub

8/8/14 setup new ubuntu vm

Ubuntu git maintainers team has a PPA

sudo add-apt-repository ppa:git-core/ppa
sudo apt-get update
sudo apt-get install git

emacs 24 ppa

sudo add-apt-repository ppa:cassou/emacs
sudo apt-get update
sudo apt-get install emacs24 emacs24-el emacs24-common-non-dfsg

replaced by?

https://launchpad.net/~ubuntu-elisp/+archive/ppa.

git checkout files from other branches into index

git checkout <branch_name> -- <paths>

install git

install emacs24

git clone dotfiles (some tweaking)

install zile, openjdk-7-jdk, maven2, tree, lein (2.2 initially),

emacs right Meta key not working: http://www.emacswiki.org/emacs/PuTTY#toc7

emacs color theme not displaying

configure test git repo build server cert, import to java keystore

curl -v -u [QID] http://[location of cert file]
sudo keytool -import -alias build_self_signed -keystore /etc/ssl/certs/java/cacerts -file foo.cert

cant figure out wget or curl from wiki page to vm due to authentication failure. Resorting to scp from desktop.

git clone Alex's test repo git clone assembla repo; test lein compile working on public key authentication w/ putty

8/19/14

ubuntu node ppa https://launchpad.net/~chris-lea/+archive/ubuntu/node.js

sudo apt-add-repository ppa:chris-lea/node.js
sudo apt-get install nodejs

(nodejs includes npm)

sudo npm install -g grunt-cli

8/28/14

rpm - list files in a package

rpm -qlp foo.rpm

use different leiningen version: just edit header of ~/bin/lein and re-run. It will install to `/.lein/self-installs/.

command line if

if id -u foo; then echo "true"; else echo "false"; fi

display the return status of most recently executed foreground command or pipeline:

echo $?

9/3/14 - rpm

build

lein do clean, with-profile rpm-generic install

install

sudo rpm -v -i target/foo.rpm

check install

rpm -q foo

uninstall

rpm -e foo

examine file metadata

rpm -qp foo.rpm --qf "%{name} %{url}\n"

see all possible metadata tags

rpm --querytags
rpm --querytags | xargs -I tag rpm -qp foo.rpm --qf "tag: %{tag}\n"

List contents of RPM package

rpm -qlp package.rpm

9/8/14

hdfs

sudo -u hdfs hadoop fs -mkdir /data
sudo -u hdfs hadoop fs -mkdir /data/sources

Copy Folders from Local Directories to HDFS Source Directory

sudo -u hdfs hadoop fs -copyFromLocal /home/bob/data/sources/aviz_datasets /data/sources

single path

sudo -u hdfs hadoop fs -put <localfile> <hdfspath>

9/13/14 putty notes

create Windows shortcut

modify properties; shortcut should read something like

"C:\Program Files (x86)\putty\putty.exe" -load foo

for terminal emacs

http://www.emacswiki.org/emacs/PuTTY

change term colors

Connection->Data->Terminal-type string = "xterm-256color"

make both 'alt' keys be 'meta' keys; regedit

Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Keyboard Layout]
"Scancode Map"=hex:00,00,00,00,00,00,00,00,03,00,00,00,1d,00,3a,00,38,00,38,e0,00,00,00,00

Explained here http://smallvoid.com/article/winnt-scancode-map.html

  • first 8 bytes are headers
  • 03 00 00 00 indicates 3 entries in map (including null entry)
  • 1d 00 3a 00 makes right Alt key work as a meta key
  • 38 00 38 e0 makes Caps Lock key another Ctrl key
  • null entry at the end

9/25/14

Gnu screen quick notes

Config file: ~/.screenrc

task binding
split M-S
close splits M-Q
rename window M-A
focus region M-tab
window list M-"
copy mode M-[

9/29/14

Needed to use eval to start ssh-agent, something to do with env variable(s) that running ssh-agent exports, and which ssh-add needs (env | grep SSH_)

eval `ssh-agent -s`
ssh-add foo

7/15 p.s.

env | grep SSH_A | awk '{print "export " $0}' > ~/ssh_agent

10/5/14

post JSON via curl

curl -X POST -H "Content-Type: application/json" -d @file.json http://hostname:8080/api/v1/foo/bar

reboot vm

sudo reboot

10/28/14

Emacs tramp mode

C-x C-f /ssh:user@hostname:/path/to/file

10/29/14

Logged in as root, create a user account for myself (RHEL)

useradd -m -s /bin/bash sbale
mkdir /home/sbale/.ssh
...
chown -R sbale:sbale /home/sbale/.ssh/
chmod 600 /home/sbale/.ssh/*
passwd -S sbale

See process using port 8083

sudo netstat -tapen | grep 8083
ps -fp [pid]

or Mac OS X

sudo lsof -n -i4TCP:8083 | grep LISTEN
sudo lsof -i -n -P | grep TCP | grep LISTEN

11/3/14

sudo hadoop fs -rm -f -r -skipTrash /user/foo

11/13/14

rejigger linux users & groups

UIDs and GIDs

id -u username
id -g username
id -G username
getent group [gid]
getent group [groupname]
getent passwd [id]
getent passwd [uname]

change user UID, GID

usermod -u [new uid] -g [new gid]
groupmod -g [new gid]

new user foo

groupadd -g 2112 foo
useradd -g 2112 -u 5150 foo

(w/o id)

groupadd foo
useradd -g foo foo

(adduser perl script, delegates to useradd)

modify user foo

groupmod -g 2112 foo
usermod -u 5150 -g 2112 foo

add user to group

usermod -a -G groupName userName

remove user

userdel foo

what groups is user member of?

groups username
id username

12/4/14

truncate lines returned by grep by piping through cut

grep whatever | cut -c1-170

12/24/14

yum package manager

yum | more
yum list installed emacs*
yum repolist
reposync
yum info hadoop-hdfs

/etc/yum.repos.d/

yum upgrade foo

yum install yum-utils
repoquery --list packagename

compare with zypper

zypper search --installed emacs*
zypper se -i emacs*
zypper info hadoop-hdfs

1/5/15

SUSE Linux equivalent to yum is yast2 or zypper

1/7/15

cider-repl-set-ns

C-c M-n

1/19/15

git push select commit to remote branch 2.3.x:

git push origin 2.3.x^^:2.3.x
git push origin 235a3sd:2.3.x

git, find commit with blarg in commit message

git log --grep="blarg"

Is a commit in branch?

Assuming a commit a21bf32 is found, see what (local and remote) branches contains commit:

git branch -a --contains a21bf32
git tag --contains a21bf32

2/10/15

show disk usage

df -h

disk usage

du -h /path/to

disk usage with grand total

du -sh /path/to

2/14/15

setting up new ec2 instance for development, quick checklist

  1. launch instance (Amazon linux)

  2. ssh ec2-user

  3. create scott user (see above)

  4. scp authorized_keys file to /home/scott/.ssh/

  5. add scott to /etc/sudoers.d/ somewhere

     echo "scott ALL=(ALL) ALL" >> /etc/sudoers.d/00_foo

  6. ssh scott

  7. yum install tmux git

  8. generate new key for github

  9. git clone stuff (dotfiles, website)

  10. build emacs from src (see above or emacs.md) a. yum install gcc autoconf automake texinfo ncurses-devel pkgconfig

  11. apache2 yum install httpd mod_ssl

  12. yum install links

  13. make user dir readable, executable chmod

  14. mod_php

  15. mysql

3/09/15

pgrep -fl foo

3/16/15

emacs tramp mode: using su to edit stuff as root

C-x C-f /su::/etc/hosts RET

get fingerprint of httpd cert (see ssl.conf)

sudo openssl x509 -fingerprint -in foo.crt

or

sudo openssl x509 -fingerprint -in foo.crt -sha256

4/6/15

deleting a git tag locally and remotely

git tag -d footag
git push --delete origin footag

4/9/15

make an existing git branch track a remote branch

git branch -u origin/foo
git branch -u origin/foo foo
git branch --set-upstream-to origin/foo

4/10/15

tmux - search pane in copy mode (similar to emacs)

Search down

Ctrl-s
[phrase]
Enter
n

Search up

Ctrl-r

4/13/15

copy-paste in tmux: in copy mode select with ctrl-space, copy with ctrl-w, paste with M-]

emacs replace-string with newline

C-q C-j return

work vm quickstart - RHEL

yum install screen tree emacs-nox

(currently version 23.1 of emacs - boo!)

echo "escape ^]]" > .screenrc
put key in ~/.ssh/

4/18/15

ec2 instance updates - wrong httpd package

http://serverfault.com/questions/570821/amazon-linux-lamp-with-php-5-5

repoquery --requires php55
yum list installed http*
sudo yum remove httpd.x86_64 httpd-tools-2.2.29-1.4.amzn1.x86_64
sudo yum install php55 httpd24

5/7/15

Git - find first instance of token STATIC_PATH in commit history:

git log -S STATIC_PATH --source --all

5/17/15

Emacs line-wrapping and fill column: see fill-column variable

M-q
M-x fill-paragraph
C-x f
M-x set-fill-column
(setq-default fill-column 80)
M-x column-number-mode

5/24/15

more on kirkwoodcoc ec2 instance...

install: php55, httpd24, mod24_ssl, mysql55, mysql55-server, php55-mysqlnd

sudo service mysqld start|status|stop

output from starting: {quote} PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER ! To do so, start the server, then issue the following commands:

/usr/bin/mysqladmin -u root password 'new-password' /usr/bin/mysqladmin -u root -h ip-172-30-0-142 password 'new-password'

Alternatively you can run: /usr/bin/mysql_secure_installation

You can start the MySQL daemon with: cd /usr ; /usr/bin/mysqld_safe &

You can test the MySQL daemon with mysql-test-run.pl cd /usr/mysql-test ; perl mysql-test-run.pl {quote}

install php mysql extension

"Your PHP installation appears to be missing the MySQL extension which is required by WordPress."

Find extensions dir:

sudo php -i | grep extension

found

extension_dir => /usr/lib64/php/5.5/modules => /usr/lib64/php/5.5/modules

install php55-mysqlnd

apache logs: /var/log/httpd/

5/25/15

Back to php 5.3, httpd 2.2

sudo yum remove httpd24 mod24_ssl php55-mysqlnd
sudo yum install httpd httpd-tools-2.2.29-1.4 mod_ssl php php-mysqlnd

5/29/15

git clone dotfiles into my existing home dir (may have to move files out of way)

git init
git remote add origin git@blahblahblah
git fetch origin
git checkout -b ubuntu-server --track origin/ubuntu-server

a good sequence to migrate to a new jumpbox virtual machine vm

  • setup ssh
  • sudo apt-get install git emacs24-nox tmux zile tree openjdk-7-jdk
  • git clone dotfiles (see above)
  • exit, relaunch tmux
  • git clone dev project(s)
  • lein self install

maven2 wasn't needed openjdk-8 isn't available via ubuntu yet w/o ppa

5/30/19

more on kirkwoodcoc on ec2...

  • done migrating back to Apache 2.2, php 3

  • permalinks broken (404), had to change AllowOverride None to AllowOverride All in

      /etc/httpd/conf/httpd.conf

    Under Directory for kirkwoodcoc

6/4/15

lein-dist

  • lein-package plugin (user pliant on GitHub)

  • lein-dist teradata plugin, defines dist, rpm tasks

  • add hooks to project.clj, e.g.:

      :hooks [leiningen.package.hooks.deploy leiningen.package.hooks.install]

  • define :package and :dist values in project.clj

  • invoke via hooked e.g. install or manually e.g.

      lein package
  lein dist
  lein rpm

6/17/15

mount the Linux-mountable share on a cloud VM:

sudo mount thehost:/vol/path/to /mnt/blah2
sudo umount /mnt/blah2

7/1/15

git export commit patch

git format-patch

apply patch

git am

7/6/15

TEZ troubleshoot log checklist

  • loom-server.log

  • hive.log

      /tmp/app.user.name/hive.log
  ${java.io.tmpdir}/${user.name}

  • JobHistory logs

      MapReduce2 service, History Server component
  tdh125m2
  /var/opt/teradata/log/hadoop-mapreduce/mapred/

  • ResourceManager and find the logs

      YARN service, Resource Manager component
  tdh125m1 (active)
  /var/opt/teradata/log/hadoop-yarn/yarn/
  ${yarn.nodemanager.log-dirs}

7/8/15

docker

docker build -f tempfile -t nugent
docker build -f tempfile -t nugent .
docker run -t -i nugent /bin/bash
docker run -t -i nugent sudo -H -u docker /bin/bash

7/10/15

problem with certs in curl, ubuntu 14:

curl: (60) error setting certificate verify locations:
  CAfile: /usr/share/centrifydc/apache/certs/ca-certs.crt
  CApath: none

More details here: http://curl.haxx.se/docs/sslcerts.html

try curl -V

temp workaround export CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

location of certs: /etc/ssl/certs/

Centrify OpenSSH replaces stock OpenSSH impl dpkg -l | grep centrify

yields matches on new jumpbox, none on old.

$ locate sshd_config
/etc/centrifydc/ssh/sshd_config
/etc/ssh/sshd_config
  • centrifydc - Centrify DirectControl infrastructure
  • adclient - centrify Active Directory support

http://community.centrify.com/t5/DirectControl-Express-for-UNIX/Curl-OpenSSL-certificate-question/td-p/15539

ldd - print shared library dependencies e.g.

ldd /usr/sbin/sshd | grep -i gssapi

startup services

update-rc.d FOO defaults
update-rc.d -f FOO defaults

service --status-all

7/14/15

clone single branch of large project

git clone --single-branch --branch release-0.5.2 [email protected]:apache/tez.git

java thread dump

kill -3 jvm-pid

7/16/15

epiphany about the purpose of shims: code may be compiled (javac) using completely different .jars|artifacts than are available at runtime (java); java classpath may be built dynamically, finding provided jars of differing versions. (Hadoop, Android, etc.)

7/17/15

running hive unit tests on the build box

  • ssh to build slave
  • switch to jenkins (assuming I'm a sudoer): sudo su jenkins
  • cd /var/lib/jenkins/workspace/temp-loom/ (git clone if necessary)
  • export WORKSPACE=/var/lib/jenkins/workspace/temp-loom
  • ./build-plugins
  • lein copylib (accomplishes same thing as lein test :jenkins)
  • ./prebuilt-hive-tests.sh

applying a patch

git am patchfile.patch

deleting a remote branch after merge

git push origin --delete foobranch
git push origin :foobranch

7/23/15

hive shell

/usr/hdp/current/hive-client/bin/hive -f hql.txt

hive warehouse, HDP

hadoop fs -ls /apps/hive/warehouse/

Cloudera (CDH)

hadoop fs -ls /user/hive/warehouse

7/28/15

docker save --output=proto-jenkins-temp.tar proto-jenkins-temp:latest
docker load --input /tmp/proto-jenkins-temp.tar

docker run --entrypoint="" -t -i -v $WORKSPACE:/home/docker/workspace proto-jenkins-lein

7/30/15

ssh-agent revisited

ssh-agent -s > foo
source foo

Now other terminals can source foo for the same env vars.

7/31/15

docker, enter running container interactively

docker exec -it [container-id] bash

rename image

docker tag [currentname]:latest [newname]:latest

remove image tag (leaving image intact)

docker rmi [tag]

8/5/15

putty - ssh to virtualbox vm via port forwarding.

  • In vm, sudo apt-get install openssh-server
  • In vm manager, settings->network, port forwarding button, add rule: ssh, TCP, host port 3022, guest port 22
  • In putty, connect to 127.0.0.1 port 3022 under connection->ssh->tunnels, add a new rule: source port: 3022 destination: 127.0.0.1:22

8/6/15

tmux, pop split pane out of window into its own window; join back to original window (window #2 in this example)

:break-pane
:join-pane -t 2

git, change remote url

git remote set-url origin [email protected]:revelytix/rsc-ls.git

8/10/15

netstat -pan | grep 10000

8/17/15

extract RPM contents w/o installing RPM

rpm2cpio foo.rpm | cpio -idmv

where cpio args are: -i = extract -d = make directories -m = preserve modification time -v = verbose

8/25/15

start virtualbox vm headless on Windows

"C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" startvm dev --type headless

also

list vms
controlvm dev poweroff

Quick Docker install - Ubuntu http://docs.docker.io/en/latest/installation/ubuntulinux/

sudo sh -c "echo deb http://get.docker.io/ubuntu docker main\
> /etc/apt/sources.list.d/docker.list"

sudo apt-get install lxc-docker

8/26/15

centrify authentication flush cache after changing password

sudo adflush

docker-related networking

apt-get install iputils-ping
  • resolveconf
  • dnsmasq
  • Ubuntu's NetworkManager

tmux change status bar color

set -g status-bg black
set -g status-fg white

8/27/15

on edge node, creating hbase table & corresponding hive external table

https://github.com/romainr/hadoop-tutorials-examples/tree/master/hbase-tables

  1. Generate column names and data with create_schemas.py. Run it with ./create_schemas.py
  2. Upload the date data /tmp/hbase-analytics.tsv to HDFS with File Browser (same path)
  3. In HBase Browser create table with
  4. Load the data into the analytics table with the HBase bulk import command.total3 , daycolumn , hourfamilies analyticsa

Useful commands

hbase shell
$ list
$ help cmd
$ create "mytable", "columnfamily1", ...
$ describe "mytable"
$ scan "mytable"
$ scan "mytable", {COLUMNS => "columnfamily1:"}
$ scan "mytable", {COLUMNS => "columnfamily1:columnname"}
$ version

hive --version
hive
$ use databasename;
$ show create table mytablename;
$ CREATE EXTERNAL TABLE analytics_total_us (key int, tot string)
ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe'
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,total:US")
TBLPROPERTIES("hbase.table.name" = "analytics");

Debugging empty Hive external table (populated table in HBase)

Jim recommends checking logs of: hive; metastore; resource manager or job history (if Hive submits a MR or Tez job)

In Ambari

  • MapReduce2 QuickLinks contain JobHistory
  • YARN QuickLinks contain ResourceManager

on tdh146m1:

tdh146m1:~ # locate tdh146m1.log
/var/opt/teradata/log/ambari-metrics-collector/hbase-ams-master-tdh146m1.log
/var/opt/teradata/log/hadoop-yarn/yarn/yarn-yarn-resourcemanager-tdh146m1.log
/var/opt/teradata/log/hadoop-yarn/yarn/yarn-yarn-timelineserver-tdh146m1.log
/var/opt/teradata/log/hadoop/hdfs/hadoop-hdfs-journalnode-tdh146m1.log
/var/opt/teradata/log/hadoop/hdfs/hadoop-hdfs-namenode-tdh146m1.log
/var/opt/teradata/log/hadoop/hdfs/hadoop-hdfs-zkfc-tdh146m1.log
/var/opt/teradata/log/hbase/hbase-hbase-master-tdh146m1.log

on tdh146m2:

tdh146m2:~ # locate tdh146m2.log
/var/opt/teradata/log/hadoop-mapreduce/mapred/mapred-mapred-historyserver-tdh146m2.log
/var/opt/teradata/log/hadoop-yarn/yarn/yarn-yarn-resourcemanager-tdh146m2.log
/var/opt/teradata/log/hadoop/hdfs/hadoop-hdfs-journalnode-tdh146m2.log
/var/opt/teradata/log/hadoop/hdfs/hadoop-hdfs-namenode-tdh146m2.log
/var/opt/teradata/log/hadoop/hdfs/hadoop-hdfs-namenode-tdh146m2.log.1
/var/opt/teradata/log/hadoop/hdfs/hadoop-hdfs-zkfc-tdh146m2.log
/var/opt/teradata/log/hbase/hbase-hbase-master-tdh146m2.log

also

/tmp/[username]/hive.log
/var/opt/teradata/log/hive/hivemetastore.log

(The verdict: user error, duh. Bad external table column mapping.)

8/28/15

manual logrotate

logrotate --force /etc/logrotate.d/loom

8/31/15

curl foo to capture and use a session cookie in bash script (simple authentication)

read -p "username: " ACCOUNT
read -s -p "pwd: " PASSWORD

export SESSION=$(curl -s -i -X POST -d "{\"username\":\"${ACCOUNT}\",\"password\":\"${PASSWORD}\"}" \
    -H "Content-type: application/json" http://${HOST}:${PORT}/api/path/to/login \
    | grep -i Set-Cookie | cut -f1 -d\; | cut -f2 -d=)

curl --cookie session-id=$SESSION -H "Content-type: application/json" ...

9/2/15

curl --manual | less

9/3/15

commit, name and tag a new image from a container:

docker commit <Container ID> <Name>:<Tag>

9/10/15

emacs search, toggle case-sensitivity: M-c

new group, add to hdfs group

groupadd foo
useradd -g foo -G hdfs foo

9/11/15

sent an e-mail from within emacs, M-x mail mode! Via smtp server.

9/14/15

temp lein test selector

:test-selectors { ...
:foo (fn [m] (and (.startsWith (str (:ns m)) "some.test.namespace") (:test-selector m)))

9/18/15

list open files: lsof

lsof -i :<port>

set of file descriptors open in a process ls -l /proc//fd

9/21/15

add user to sudoers in SLES

visudo

(edits /etc/sudoers file)

9/25/15

Python HTTP server from current working directory

python -m SimpleHTTPServer [port]

Windows hosts file:

systemroot\System32\Drivers\Etc\hosts

9/29/15

chkconfig stuff

chkconfig --del loom

chkconfig --level 345 loom on

chkconfig --list loom
loom            0:off   1:off   2:off   3:on    4:on    5:on    6:off

find /etc/rc.d/ | grep loom

9/30/15

install pip using python easy_install; aws cli using pip

python --version
sudo easy_install pip
sudo pip install awscli

upload to s3 bucket

aws configure
aws s3 ls
aws s3 ls s3://bucket-name/path/to/

VirtualBoxHeadless configure, open a .ova image

apt-get install virtualbox, virtualbox-dkms
VBoxManage import foo.ovf
VBoxManage list vms

set up port-forwarding rule:

VBoxManage modifyvm "foo-vm" --natpf1 "ssh,tcp,127.0.0.1,2222,,22"

deactivate hardware virtualization acceleration

VBoxManage modifyvm "foo-vm" --hwvirtex off

Set password for VNC connection. (If Oracle's proprietary extension pack isn't installed, then we are limited to VNC connection. Otherwise the better VRDE protocol is enabled. Windows can connect via native mstsc client.)

vboxmanage modifyvm "foo-vm" --vrdeproperty VNCPassword=nuuuge

VBoxManage showvminfo "foo-vm"
VBoxManage controlvm "foo-vm" poweroff

flag --vrde controls vrde|vnc server

VBoxHeadless -startvm "foo-vm" --vrde off

D'oh: https://www.virtualbox.org/manual/ch03.html#intro-64bitguests

10/6/15

aws s3 cli

see ACL for bucket

aws s3api get-object-acl --bucket foo-bucket --key path/to/
aws s3api get-object-acl --bucket foo-bucket --key path/to/filename

set ACL, grant permissions

aws s3api set-object-acl --bucket foo-bucket --key path/to/filename --grant-read 'uri="http://acs.amazonaws.com/groups/global/AllUsers",uri="http://acs.amazonaws.com/groups/global/AuthenticatedUsers"' --grant-full-control 'id="longhexuuid"'

10/15/15-21

Leiningen upgrade

  • Currently on 2.0.0
  • want 2.5.1 (latest still using Clojure 1.6)
  • 2.4.3 works w/o mods
  • plugin [lein-pprint "1.1.1"]
  • 2.0.0 -> 2.4.3
    • lein included version of cemerick's pomegranate goes from 0.0.13 to 0.3.0
    • upgrade lein-package from 2.0.1 to 2.1.0
  • DEBUG=y lein foo prints lein debug output
  • run leiningen locally
    • cd [project]/leiningen-core
    • lein bootstrap
    • then invoke [project]/bin/lein

questions

  • reduce compilations
    • lein-package plugin - reuse compilations?
    • project.clj - change :prep-tasks ?

inside docker container, hadoop pseudo-cluster, found logs at:

tar czvf hadoop-logs.tar.gz /var/log/hadoop-hdfs /var/log/mysql /var/log/hive /var/log/hadoop-0.20-mapreduce ...

10/29/15

update node PPA

sudo apt-add-repository -r ppa:chris-lea/node.js
curl --silent --location https://deb.nodesource.com/setup_4.x | sudo bash -
sudo apt-get install --yes nodejs

11/2/15

Node

node -v
v0.12.7
npm -v
2.11.3

sudo apt-get install g++

# Uninstall everything, start from clean slate
sudo apt-add-repository -r https://deb.nodesource.com/node_4.x/trusty/main
sudo rm /etc/apt/sources.list.d/nodesource.list*
rm -fr node_modules/
rm -fr ~/.npm
sudo apt-get remove -y nodejs
sudo apt-get autoremove
sudo apt-get clean
sudo apt-get update
sudo rm -fr /usr/lib/node_modules

# Install node 0.12, npm 2.11.3
curl -sL https://deb.nodesource.com/setup_0.12 | sudo -E bash -
sudo apt-get install -y nodejs

# Install gulp globally
sudo npm install -g gulp

# And finally...
npm install

11/3/15

Have .profile create, or reuse, ~/.sshagent file

Clean git working directory of untracked files

git clean -d -x --force --quiet

11/5/15

emacs regex with whitespace

\s-stringtomatch

11/10/15

M-x vc-git-grep

11/11/15

bash one-liner to see npm package name, version for all packages in node_modules directory

for mo in `ls node_modules`; do cat "node_modules/$mo/package.json" | python -c 'import sys, json; packg=json.load(sys.stdin); print packg["name"], packg["version"]'; done

diff -r <dir1> <dir2>

11/16/15

~/dev$ diff -r ui/node_modules/gulp-minify-css/ ui2/node_modules/gulp-minify-css/ | less

11/17/15

sigterm, from signal.h, for kill command

  • 1 is SIGHUP, terminal hangup
  • 2 is SIGINT, interrupt, equivalent to Ctrl-c
  • 3 is SIGQUIT (use to get Java thread dump)
  • 9 is SIGKILL
  • 15 is SIGTERM, the default

11/30/15

dired, omit node_modules

hashsum on unix, windows

  • unix

      shasum -a 256  path/to/file

  • windows

      CertUtil -hashfile /path/to/file SHA256

12/17/15

zipping and tar'ring

tar czvf foo.tar.gz dir1/ dir2/
tar xvf foo.tar.gz
tar tf foo.tar.gz
zip -r foo.zip dir1/
unzip foo.zip
unzip -l foo.zip

1/23/16

install Java 8; update java alternative to JDK 1.8 on amazon linux ec2 instance

yum search java
sudo yum install java-1.8.0-openjdk-headless.x86_64
ll /usr/lib/jvm/
alternatives --usage | less
alternatives --display java
sudo alternatives --set java /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java
java -version

Mac OS Yosemite hosts file: /private/etc/hosts

2/2/16

First two days at DataStax

Setting up MacBook Pro

  • had to install XCode to get git (should have just installed XCode-cli but oh well)

  • new ssh key, add to GitHub keys, git clone stuff

  • onboarding script using Ansible Playbook (running into Ansible bugs)

      https://github.com/ansible/ansible/issues/13728

  • Ansible installed HomeBrew

  • I used brew to install emacs

      brew install emacs --with-cocoa
  brew linkapps emacs

  • on emacs startup, initially got a compilation error - technomancy starter kit dependency on dash 2.12.1 (had to install that version by hand using M-x package-list-packages)

  • Lots of hand-tweaking of dotfiles

2/3/16

Day 3 - Wed

  • CCM 2.1.3 installed via pip

  • 3-node DSE 4.8.0 cluster

      ccm create dse-480-1 --dse --dse-username=your_username --dse-password=your_password -v 4.8.0 -n 3

where uname/password is from download site

OS X keybindings

Struggling to find balance between command key and tmux|terminal Meta-keybindings I'm used to

Got it!

  • Leave stock global keybindings alone
  • in iTerm preferences
    • Keys: Change right command key to option
    • Keys: +, then add cmd-whatever combos to ignore
    • Profiles->Keys: Option should send +Esc

Almost all the cmd keybindings I care about (cmd-w, cmd-q, cmd-c, cmd-v, cmd-x, cmd-tab) happen to be left command key chords. Likewise, almost all the tmux and terminal meta-key combos I care about happen to be right meta key chords (M-d, M-f, M-v). Notable exceptions: M-<, M->. update 5/26/21 wat? aforementioned cmd keybindings are right command key chords. [later] oh right 9/20/17

2/4/16 - Thu

telnet host port - all-purpose check for any process listening on a port? learn: nmap, netcat

2/5/16

Paste into|from OS X Gnu Emacs from|to clipboard

M-x clipboard-yank
M-x clipboard-kill-region

See also http://www.emacswiki.org/emacs/CopyAndPaste

Installing python deps

Trying to pip install python-ldap, Missing sasl.h file; had to do this:

xcrun --show-sdk-path
sudo ln -s <the_path_from_above_command>/usr/include /usr/include

...but looks like a better way would be

pip install python-ldap \
  --global-option=build_ext \
  --global-option="-I$(xcrun --show-sdk-path)/usr/include/sasl"

Trying to pip install -I pyOpenSSL=0.13, got compilation error

OpenSSL/crypto/crl.c:6:23: error: static declaration of 'X509_REVOKED_dup' follows non-static declaration

solution is to install 0.14

sudo pip install -r requirements.txt

LDFLAGS:  -L/usr/local/opt/openssl/lib
CPPFLAGS: -I/usr/local/opt/openssl/include

you should be putting those flags somewhere in setup.py IIRC

2/8/16

Generally, switching branches should be accompanied by invoking ant

alternatively

npm prune, install, update

http://localhost:8888/opscenter/js/bower_components/util/doh/runner.html?testModule=ripcord.tests.all&boot=../../../ripcord-compiled/tests/dojoconfig.js,../../dojo/dojo.js http://localhost:8888/opscenter/js/bower_components/util/doh/runner.html?testModule=ripcord.tests.unit.widgets.backups.snapshotspager&boot=../../../ripcord-compiled/tests/dojoconfig.js,../../dojo/dojo.js

emoticons: tumbleweed, waiting, upvote|downvote, disapproval, thisisfine, oldschool, cerealspit, derp, giggity, wat, lol, lolwut, paddlin, shrug, awyeah, iseewhatyoudidthere, success, tableflip, indeed (monocle), whynotboth, puke (rainbow), pokerface, badpokerface, salute

2/10/16

EC2, GCE, Azure

python issue workaround

Use homebrew to install a (keg only) version of openssl that pyOpenSSL 0.13 will compile with:

brew install homebrew/versions/openssl101

Install pyOpenSSL, compiling against this version of openssl

LDFLAGS=-L/usr/local/opt/openssl101/lib CPPFLAGS=-I/usr/local/opt/openssl101/include pip install -I pyOpenSSL==0.13

2/11/16

UI build/config:

  • ant build.xml - ant ui runs npm install, gulp
  • node npm package.json => bower, gulp
  • Gulp Gulpfile.js - runs bower build for dependencies; sass; babel; tests; ...
  • Bower bower.json => dojo and related
  • DOJO build.profile.js

Show all ignored and untracked files

git ls-files --ignored --others --exclude-standard --directory

run python http server from ripcord/opscenterd/src/js, load http://localhost:8888/bower_components/util/doh/runner.html?testModule=... or is it...? http://localhost:8888/ripcord/tests/bootstrap.html#ripcord/tests/foo/bar

2/13/16

cljs

in hello world terminal:

java -cp cljs.jar:src clojure.main repl.clj

in cljs src terminal:

./script/build
lein repl :headless :port 2112

install mvn on Amazon Linux

sudo wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo \
    -O /etc/yum.repos.d/epel-apache-maven.repo
sudo sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo
sudo yum update
yum search apache-maven
yum info apache-maven
sudo yum install -y apache-maven
mvn --version

2/17/16

loopback

sudo ifconfig lo0 alias 127.0.0.x up

2/23/16

ui, when switching between branches

npm prune && npm install && npm update

and/or

rm -fr ./node_modules
npm cache clean && npm install

2/25/16

python pretty print JSON awesomeness

cat some.json | python -m json.tool

anvil (alternative to python simple http server)

2/26/16

homebrew routine

brew update
brew outdated
brew upgrade [foo]

also

brew doctor
brew --cache

2/29/16

JS typechecking:

  • FaceBook Flow
  • Term (sp?) - typechecking against JSDoc

3/3/16

Today's coolness:

  • wrote a python script to examine an array of JSON returned from a RESTful API endpoint

  • figured out how to send multiple params in a cURL cmd (was missing quotes, therefore shell was backgrounding the process because of the ampersand)

      curl -sS 'http://whatever:8888/blahblah?foo=bar&goo=gar' | ./awesome-python-script.py

  • figured out how to map over a list in python

  • in JS, using dojo, did some map, reduce

  • debugged a unit test in Chrome, with breakpoints - figured out how to look for loading sources

3/3/16

using opsc from src, cluster is at local/clusters/Test_Cluster.conf

3/8/16

ccm node1 setworkload solr

create solr core w/ ccm

http://docs.datastax.com/en/latest-dse/datastax_enterprise/srch/srchCreatCore.html

ccm node1 dsetool create_core "OpsCenter".backup_reports generateResources=true

3/9/16

mac os x | BSD locate and updatedb

man locate
/usr/libexec/locate.updatedb

alternatively,

brew install findutils

to get gnu locate and updatedb

3/10/16

prop|config files

  • cassandra.yaml
  • cassandra-rackdc.properties

opsc ports: 7100, 9042

C* basics:

keyspace = collection of tables

  • replication factor, set per DC

replication factor (RF; writes; fixed); contrast w/ consistency level (CL; reads and writes; changes)

  • SimpleStrategy - single DC
  • NetworkTopologyStrategy - mult DC's

token range

single token node (node maps to final token in its range); contrast w/ vnodes (num_tokens in cassandra.yaml)

partitioner - hashes tokens from designated values in rows being added

  • partition key: piece of primary key

hinted handoff - recovery for writes targeting offline node

  • stored in system.hints keyspace

consistency level (CL)

  • writes - how many nodes must respond write was made durable

  • reads - how many nodes must respond with latest copy of data

  • quorum aka majority = RF / 2 + 1

  • immediate consistency aka ALL - highest latency

      ccm [node] dsetool [cmd]
  ccm [node] status - nodetool status
  ccm [node] cqlsh
  ccm [node] nodetool ring
  ccm [node] nodetool getendpoints [keyspace] [table] [partition key value(s)]

write path

  • Memtables - in-memory CQL tables, indexes
  • CommitLog - append-only
  • SSTables aka "Sorted String Tables" - Memtable snapshots flushed to disk, clearing heap - never changed, supercedes previous
  • Compaction - periodic, of SSTables

bin/sstable2json

3/11/16

simple keyspace

cqlsh> create keyspace foobar WITH replication = {'class': 'SimpleStrategy','replication_factor':2};
cqlsh> use foobar;
cqlsh:foobar> create table peeps (id int primary key, name text, address text, phone text);
cqlsh:foobar> insert into peeps (id, name, address, phone) values ( 1, 'Scott', 'MO', '867-5309');
cqlsh:foobar> select * from peeps;

 id | address | name  | phone
----+---------+-------+----------
  1 |      MO | Scott | 867-5309

(1 rows)
cqlsh:foobar> select token(id) from peeps;

 token(id)
----------------------
 -4069959284402364209

(1 rows)
cqlsh:foobar> consistency;
Current consistency level is ONE.
cqlsh:foobar> update peeps set phone='555-867-5309' where id=1;
cqlsh:foobar> select * from peeps;

 id | address | name  | phone
----+---------+-------+--------------
  1 |      MO | Scott | 555-867-5309

(1 rows)
cqlsh:foobar>

and

ccm node1 nodetool getendpoints foobar peeps 1;

(later)

cqlsh:foobar> alter table peeps add loudness int;
cqlsh:foobar> update peeps set loudness=3 where id=1;
cqlsh:foobar> update peeps set loudness=11 where id=2;
cqlsh:foobar> insert into peeps (id, name, address, phone, loudness) values ( 3, 'M. Loaf', 'WI', '867-3333', 8);
cqlsh:foobar> create materialized view loudest as select id, name, loudness from peeps where
name is not null and loudness is not null primary key (loudness, id) with clustering order by
(loudness desc);

3/14/16

pip uninstall, install diff version

sudo pip uninstall foo; sudo pip install foo==1.0.1

cache at

~/Library/Caches/

3/16/16

TIL S->|< is sufficient to page up|down in less. (Am used to emacs S-M->|< for that)

3/23/16

ccm-create DSE 5.0 nightly

pushd ${TMPDIR:-/tmp/}; curl --insecure --location https://[qa-uname]:[pwd]@downloads-qa.datastax.com/enterprise/dse-5.0.0-bin.tar.gz | tar xv; popd
ccm create new-hotness --dse -n 3 --install-dir=${TMPDIR:-/tmp/}dse-5.0.0

lein, cider, nrepl...here we go again

leiningen

  • version: 2.5.2

cider-nrepl

  • version: upgrading from 0.8.2 to 0.11.0
  • 0.11.0 requires clj 1.7+ and lein 2.5.2+
  • in ~/.lein/profiles.clj

cider

  • version: upgrading from 0.8.2 to 0.11.0 (marmalade)
  • an emacs mode, package-installed via melpa, marmalade or whatnot
  • in ~/.emacs.d/

Getting this when I M-x cider-connect:

;; Connected to nREPL server running on port 2112 on host localhost - nrepl://localhost:2112
;; CIDER 0.11.0 (Bulgaria), nREPL 0.2.10
;; Clojure 1.7.0, Java 1.8.0_65
...
WARNING: CIDER requires nREPL 0.2.12 (or newer) to work properly

this is a clue...

~/dev/ripcord/agent $ lein do clean, compile, repl :headless :port 2112
Retrieving cider/cider-nrepl/0.11.0/cider-nrepl-0.11.0.pom from clojars
Retrieving org/clojure/tools.nrepl/0.2.12/tools.nrepl-0.2.12.pom from central
Retrieving org/clojure/tools.nrepl/0.2.12/tools.nrepl-0.2.12.jar from central
Retrieving cider/cider-nrepl/0.11.0/cider-nrepl-0.11.0.jar from clojars
nREPL server started on port 2112 on host 127.0.0.1 - nrepl://127.0.0.1:2112

the answer: technomancy/leiningen#2072

3/24/16

ns-unmap a namespace's interned symbols

(map (comp (partial ns-unmap *ns*) first) (ns-interns *ns*))

3/25/16

Cassper Guard Weiner Dog

3/28/16

TIL alternative to

git lg --stat [commit range]

which is

git diff --name-only [commit range]

3/29/16

cqlsh> describe keyspaces;

3/30/16

CCM error

Error opening zip file or JAR manifest missing : /Users/scottbale/.ccm/new-hotness/node3/resources/cassandra/bin/../../../resources/cassandra/lib/jamm-0.3.0.jar

Solution

This is all from ~/.ccm/clustername/node1:

ln -s ~/.ccm/repository/4.8.0/lib lib
ln -s ~/.ccm/repository/4.8.0/resources/cassandra/lib resources/cassandra/lib
ln -s ~/.ccm/repository/4.8.0/resources/dse/lib resources/dse/lib
ln -s ~/.ccm/repository/4.8.0/resources/hadoop/lib resources/hadoop/lib

4/1/16

cqlsh:OpsCenter> select event_time, blobAsBigInt(timestampAsBlob(event_time)), type from backup_reports where week='201614' ;
select backup_id, event_time, blobAsBigInt(timestampAsBlob(event_time)) from backup_reports where week='201613' and event_time = '2016-03-30 20:35:24.250+0000' ;

4/4/16

quick checklist to wipe opsc clean

  1. remove logs
  2. drop opsc keyspace
  3. revert cluster conf

drop opsc keyspace, in CQLSH

cqlsh> describe keyspaces;
cqlsh> drop keyspace "OpsCenter";

4/7/16

Per Phil, check out logging.py

4/14/16

TIL not to trust jps

~/.ccm/dse-perf-work/node1/resources/cassandra/conf/cassandra-env.sh

comment out # http://www.evanjones.ca/jvm-mmap-pause.html JVM_OPTS="$JVM_OPTS -XX:+PerfDisableSharedMem"

https://issues.apache.org/jira/browse/CASSANDRA-9483

The default JVM flag -XX:+PerfDisableSharedMem will cause the following tools JVM to stop working. jps, jstack, jinfo, jmc, jcmd as well as 3rd party tools like Jolokia. If you wish to use these tools you can comment this flag out in cassandra-env.{sh,ps1}

opsc on-server backup dir: snapshots directory below the c* data dir (/var/lib/cassandra/data by default)

  • for my ccm cluster, they're in ~/.ccm/[cluster-name]/nodeN/data0/[long-ascii-id]/snapshots/
  • see also backup_staging_dir and tmp_dir properties of agent agent/local/address.yaml
  • the above overriden by runmultiple.py in agent/local/agentN.yaml

4/18/16

OPSC-8349 notes:

  • commit log stomp channel: /commitlog/status
    • incoming: Agents.py, routeIncoming
    • outgoing: common.clj, update-commitlog-status

OPSCD

  • logging via twisted
  • stomp via StompFactory of morbid (located in opscenterd/lib/py/morbid/); referenced by OrbitedService
  • Agents.py - only 1 code route that results in a call to cleanupPIT with a non
    • see routeIncoming for dispatching
    • see processNodeDetail for example of trace logging
    • processCommitLogStatus has very little logging
      • lack of existing log statements indicates status must be running, destination ON_SERVER

How do you turn up twisted logging to trace?

  • in master, via logback.xml
  • in 5.2.x, via local\opscenterd.conf

4/21/16

To enable commitlog archiving on local ccm agents, had to symlink the archive_commitlog.sh.template file:

ln -s agent/pkg/bin/archive_commitlog.sh.template agent/bin/

fswatch - detect changes to directory tree:

brew install fswatch
fswatch -xt -e ".*\.log" ${TMPDIR:-/tmp/}dse-5.0.0 ripcord/opscenterd/local/ ripcord/agent/local/ ripcord/agent/tmp1/ ripcord/agent/tmp2/ ripcord/agent/tmp3/ ripcord/agent/backups1/ ripcord/agent/backups2/ ripcord/agent/backups3/  ~/.ccm/borscht/ /tmp/ >> fswatch.log 2>&1

4/22/16

More OPSC_8349 notes

sub-namespaces under src/opsagent/backups/, ordered roughly by dependency:

common* entities, staging* throttler destinations pit*, snapshots coordinator, restore cleanup

4/27/16

merge script:

/bin/merge.py master foo-branch

4/28/16

curl cmds for opsc backups, job schedules, requests:

create scheduled backup

curl -sSX POST 'http://localhost:8888/Test_Cluster/job-schedules' --data '{"first_run_date":"2016-04-27","first_run_time":"15:45:00","timezone":"GMT","interval":5,"interval_unit":"minutes","job_params":{"type":"backup","keyspaces":["foobar"],"cleanup_age":30,"cleanup_age_unit":"days","destinations":{}}}' 

curl -sSX POST 'http://localhost:8888/Test_Cluster/backups/destinations'  --data '[]'

job schedule (or DELETE)

curl -sS 'http://localhost:8888/Test_Cluster/job-schedules/172ce740-2319-4407-a81f-94087677836a'

status of request

curl -sS 'http://localhost:8888/request/5ee7a158-cebe-4424-8b53-6a0e31152277/status'

delete backup

curl -sSX DELETE 'http://localhost:8888/Test_Cluster/backups?tag=opscenter_b320e60d-48a6-4221-a1ff-5cea088a04d2_2016-04-27-21-20-00-UTC&destination=OPSC_ON_SERVER&remove_destination=Delete%20Backup%20Data'

backups, backup activity

curl -sS 'http://localhost:8888/Test_Cluster/backups/$keyspace'
curl -sS 'http://localhost:8888/Test_Cluster/backup-activity?count=$N'
curl -sS 'http://localhost:8888/Test_Cluster/backup-activity/full_status?week=201618&event_time=1462462500&backup_id=opscenter_ddbd74f0-4152-4dbd-92e8-f7f76a7eb4b2_2016-05-05-15-35-00-UTC&type=backup&destination=OPSC_ON_SERVER'

run backup immediately

curl -sS http://localhost:8888/Test_Cluster/backups/run --data '{"keyspaces":["foobar"],"destinations":{"b1ef74a5f4f440208aad085f4e490449":{}}}'

create destination

curl -sS -X POST 'http://localhost:8888/Test_Cluster/backups/destinations' -d '{"provider":"s3","path":"scottbale","access_key":"TODO","access_secret":"TODO"}'
time curl -sS 'http://localhost:8888/Test_Cluster/backups?list_all=1&destination=b75a4302aed24920a19bef6d07b2f585'

restore from backup

curl -sS -X POST 'http://localhost:8888/Test_Cluster/backups/restore/opscenter_fbe75dd3-dad7-4d62-9365-24cb716c2903_2016-06-15-23-19-00-UTC/foobar' -d '{"destination":"bba55a1cbd964b9081e1be4fdf9d5f72"}'

5/3/16

What gulp tasks are available?

~/dev/ripcord/opscenterd $ ./node_modules/.bin/gulp

5/6/16

Gulp test

~/dev/ripcord/opscenterd $ ./node_modules/.bin/gulp test --suite ripcord/tests/unit/widgets/foo/barwidget

KMIP = offsite key storage (in DSE context)

https://en.wikipedia.org/wiki/Key_Management_Interoperability_Protocol https://docs.datastax.com/en/latest-opsc/opsc/configure/opscConfigKMIPalert.html

5/9/16

python unit tests logged at opscenterd/tests/test_run.log

5/10/16

Python repl, you can either do:

$ python

to get a python repl or

$ ./opscenterd/bin/jython

to start a jython repl

5/17/16

ctool, part of automaton,

  • ~/.automaton.conf
  • git clone github.com/riptano/automaton
  • export PYTHONPATH="/Users/scottbale/dev/automaton":${PYTHONPATH}
  • export PATH="$PATH:/Users/scottbale/dev/automaton/bin"
  • login to openstack to see provisioning - use ldap credentials

technically ctool is the commandline interface to automaton...usually referred to synonymously

 pip install python-openstackclient

python virtualenv https://virtualenv.readthedocs.org/en/latest/

5/18/16

More on openstack:

ignore custom images, dev stuff from https://github.com/riptano/ripcord/tree/master/opscenterd/tests/automaton/tests#local-usage

ssh public key fingerprint

ssh-keygen -lf /path/to/key.pub

cmds: openstack server list, nova host-list, keystone user-list

5/19/16

AFTs

  • ec2 region is hardcoded in AFTs

  • When running AFTs use caution in syntax:

      test_file.py:TestClass.test_case

LCM = LifeCycle Manager (code-named Spock) DSE Multi-Instance (formerly known as dense nodes)

mental queue

  • jabber (distributed consensus) vs jmx, stomp, etc. - can opscd leverage?
  • drivers
  • rollups

Andy Tolberg - driver team - JobCreator project

5/20/16

ec2 instances powered off overnight - timeout? AFT rerun created new instances rather than starting up prev ones

Q2 goals: DSE - self replication, spark

JIRA comments

scp patches

  • compile class

      ./bin/jython
  >>> import py_compile
  >>> py_compile.compile('./src/path/to/Foo.py', '/path/to/Foo$py.class');

  • scp class to target

      ctool scp2 scottbale_opsc Foo\$py.class 0:/home/automaton/

  • on target, copy to /usr/share/opscenter/jython/Lib/site-packages/opscenterd

  • restart

      sudo service opscenterd restart

  • opscd log

      sudo tail -f /var/log/opscenter/opscenterd.log

5/23/16

AFT: Found agent logs at ~/dse0_agent/agent/log/agent.log

misc recent commands:

./run.sh test_dense_nodes.py:TestDenseNodesRestart
tail -f ~/dev/ripcord/opscenterd/tests/automaton/tests/testrun.log
./bin/opscenter -f > log/opscd.log 2>&1
for i in {1..5}; do curl 'http://ec2-54-153-0-232.us-west-1.compute.amazonaws.com:8888/scottbalez/diagnostics.tar.gz' > "diagnostics_B$i.tar.gz"; done
ctool scp2 scottbalez 1:/home/automaton/dse1_agent/agent/log/agent.log ../agent11.log

Bret's VBox setup notes

  • network
    • adapter 1 - attached to: NAT (I should use Bridge)
    • adapter 2 -
  • dense node cluster network
    • adapter 1 - attached to: Internal Network (descriptive string)
    • adapter 2 - ditto

see OPSC-7542 notes

  • address.yaml attachments build agent tarball, scp to nodes

add node scripts

 /etc/dse-[name]/serverconfig/

5/25/16

VirtualBox network mode options

  • NAT - Network Address Translation

5/31/16

OWASP Webgoat project

  • entity encoder API

XSS - server vulnerabilities

  • soln: validating input, encoding output

5 common HTML contexts

  • Element content: <div>RepeatAfterMe</div>
  • Attribute value: <div id="RepeatAfterMe">
  • JavaScript: <script>var a = "RepeatAfterMe";
  • CSS: <div style="RepeatAfterMe" />
  • URL Attribute value: <a href="RepeatAfterMe">

CSRF aka Sea-Surf - browser vulnerabilities

  • soln: make requests unpredictable|hard to forge via secret token in conversational state

fixed flyspell

  • brew install aspell --with-lang-en
  • tweak init.el

emacs things to check out: spacemacs, helm, projectile

6/1/16

ECS - EC2 Container Service

6/2/16

Here we go again...Getting this when I M-x cider-connect:

;; Connected to nREPL server - nrepl://localhost:2112
;; CIDER 0.12.0 (Seattle), nREPL 0.2.12
;; Clojure 1.7.0, Java 1.8.0_65
...
WARNING: CIDER's version (0.12.0) does not match cider-nrepl's version (nil). Things will break!

I should have gotten this straight long ago:

  • CIDER - package|mode for Emacs - successor to nrepl.el - clojure-emacs org on github
  • nrepl - aka tools.nrepl, Clojure networked REPL - clojure org on github
  • cider-nrepl - "A collection of nREPL middleware designed to enhance CIDER." - clojure-emacs org on github

In project.clj, upgraded tools.nrepl to 0.12.0 to match Emacs package cider.el version. That seemed to fix it! (Update 11/22: I don't understand what I meant by this! Which project.clj? Presumably agent, but it currently uses version 0.2.3. What about version of tools.nrepl in ~/.lein/profiles.clj? Currently 0.2.12. I think what I meant was, version of cider-nrepl specified in ~/.lein/profiles.clj has to match version of cider emacs package.)

Cider quit C-c C-q

Clojure mode, reindent selected region C-M-\ beginning of defun C-M-a reindent defun M-q

6/8/16

python yield, generators, iterators, https://jeffknupp.com/blog/2013/04/07/improve-your-python-yield-and-generators-explained/

Twisted supplies two decorators: defer.deferredGenerator and defer.inlineCallbacks http://blog.mekk.waw.pl/archives/14-Twisted-inlineCallbacks-and-deferredGenerator.html

AFT opscd email conf

/etc/opscenter/event-plugins/email.conf

6/9/16

google smtp: smtp.gmail.com:465 requires e-mail address and one-time password (if 2FA is enabled)

smtpd

python -m smtpd -n -d -c DebuggingServer localhost:8025

see also: https://mailcatcher.me/, https://nilhcem.github.io/FakeSMTP/

for ctool openstack instances, you need to install ruby 2.3 and libsqlite3-dev in order to install mailcatcher...

mailcatcher --ip 0.0.0.0 --smtp-port 8025 --http-port 8080 -f -v

6/15/16

To repair aws cli, uninstalling botocore-1.2.11 and installing 1.4.4 - I recall I had to downgrade to make AFT work

6/20/16

M-x cider-repl-clear-buffer

used to be bound to C-c M-o I think?

6/21/16

ps stuff, wraps the entire process string into less

 ps -ex [PID] | less

Solr again

Create ccm cluster w/ solr workload

ccm create solar-4 --dse -n 1 -v 4.8.0
ccm node1 setworkload solr
ccm start

create foobar, peeps via cqlsh as per above, then

ccm node1 dsetool create_core foobar.peeps generateResources=true

make sure numstrings.py is in your PYTHONPATH

python ~/create_solr_core.py --count 100

Oh, you'll also need the requests lib for Python... don't know if that's included with ripcord or not. http://docs.python-requests.org/en/master/

6/22/16

git rebase --onto origin/master origin/6.0.0-dev 6.0.0-scott.bale

6/24/16

My current command to invoke lein repl - need to make an alias?

lein with-profile +with-logging repl :headless :port 2112

s3

aws s3 help
aws s3 ls scottbale/snapshots/
aws s3 ls s3://scottbale/snapshots/
aws s3 rm help
aws s3 rm --dryrun --recursive s3://scottbale/snapshots/

7/5/16

automaton setup

~/dev/automaton $ python setup.py build install

repair service (opsc); repairs (Cassandra)

  • merkle tree

  • built during compaction, "validation compaction"

      curl -O -v --user [email protected]:password -L https://downloads.datastax.com/enterprise/dse-5.0.0-bin.tar.gz 
  curl -O -v --user [email protected]:password -L https://downloads.datastax.com/enterprise/opscenter-6.1.0.tar.gz

    (Was able to load https://downloads.datastax.com/enterprise/ in browser, prompted for ds academy credentials)

show emacs line numbers

M-x linum-mode

increase|decrease text size

C-x C-+|-

7/14/16

Where|how is version of Twisted specified|stored?

/opscenterd/lib/py/twisted/

See _version.py

linux container: kubernetes, mezos (mezosphere)

  • Google uses kubernetes
  • Wal-Mart uses mezosphere
  • Waze (sp?) map app (Google bought, POC phase) Apple, Fed-Ex, Waze, Nike, Kaiser-Permanente, Chase,

7/21/16

enable stdout in twisted trial tests by commenting out: https://github.com/riptano/ripcord/blob/master/opscenterd/src/opscenterd/Logging.py#L296

def installLogbackPublisherToTwisted():
#    remove_startup_observer()

8/11/16

TODO support questions

  • how does opsc logging work
  • how is opsc installed
  • opsc encryption|SSL

security glossary

  • DES
  • symmetric encryption
  • assymetric encryption
  • cacerts - a Java truststore file
  • keystore
  • truststore
  • kerberos
  • TLS/SSL
  • X.509
  • PEM - Privacy-Enhanced Mail, an ASCII encoding of DER using Base64 encoding - a PEM file is a text file of base64 encoded strings
  • DER - Distinguished Encoding Rules, subset of BER with one way encoding. X.509 relies on DER.
  • Java JCE
  • Java JKS - Java KeyStore file format
  • bcrypt
  • SHA
  • AES
  • des3
  • BouncyCastle BKS
  • PKSC 12 - archive file format for storing multiple crytographic objects - alternative to PEM
  • CA - Certificate Authority

8/12/16

~/dev $ mv ripcord/opscenterd/node_modules .node_modules/5.2.x/
~/dev $ mv .node_modules/5.2.x/node_modules ripcord/opscenterd/
~/dev $ mv ripcord/opscenterd/node_modules .node_modules/5.2.x/
~/dev $ mv .node_modules/6.0.x/node_modules ripcord/opscenterd/

OPSC-9999

Alfred App on OS X

netstat -lnt

.cert files are just text with keys in them

opscenter.conf

ssl_keyfile=Cass...-key.pem
ssl_certfile=2-Cass...cert.crt

3-Cass...cert.crt is ordered differently

emacs regex foo word boundary

\bfoo\b

8/15/16

SSL

openssl x509 -text -noout -in [foo.cer|foo.pem|foo-cert-chain-pem-encoded.crt]
openssl x509 -inform pem -in nuge.pem -pubkey -noout
keytool -printcert -v -file foo.crt

8/16/16

brew install nmap
nmap -p 8888 127.0.0.1

"open" means there is a service listening. "closed" means that you communicated successfully with the port, and nothing was listening (the OS itself responded with a RST packet, meaning I'm here but no process is listening on the port). "filtered" means that you didn't get a response at all. This can mean many things, but shouldn't happen for localhost testing.

nmap -sV -p 8888 127.0.0.1

8/19/16

Open up Chrome Settings > Show advanced settings > HTTPS/SSL > Manage Certificates

8/25/16

http://docs.datastax.com/en/opscenter/6.0/api/docs/backups.html#method-delete-a-destination http://docs.datastax.com/en/opscenter/6.0/api/docs/backups.html#method-add-or-update-a-destination http://docs.datastax.com/en/opscenter/6.0/api/docs/schedules.html#method-update-scheduled-job

8/26/16

brew install p7zip
7z x foo.7z

8/27/16

https://support.datastax.com/hc/en-us/articles/212109266

8/29/16

Keep getting these and similar when trying to switch between 5.2.x-based and 6.0.x-based branches

fatal: failed to read object 807068e074bdf5e27bd6afbd22ab75509a2ccd33: Interrupted system call
fatal: failed to read object dacd945d73bad89c1dd32ce2aaa788057c3a5ac5: Interrupted system call

9/1/16

clean up git remote branches, and local branches no longer tracking remotes

git remote prune origin
git branch -vv
git branch -a | grep 8xxx
git branch -a | grep 8xxx | xargs git branch -D

9/7/16

generate doc

~/dev/ripcord/doc/api $ sudo pip install -U -r requirements.txt
~/dev/ripcord/doc/api $ make html-sphinxdoc

EDN (Extensible Data Notation) file

netty.io

line-by-line add chunks to tree

git add --patch foo

9/13/16

riptano/jython - github.com/jythontools/jython

  • update master
  • cherry-pick into ripcord-master

9/16/16

brew install findutils
sudo gupdatedb
glocate foo

This took me way too long to figure out:

glocate automaton | grep -v "/Users/scottbale/dev/"

9/19/16

My copy of automaton is messed up, I can only seem to run it in-place (i.e. not install it).

~/dev/automaton $ python ./setup.py build
~/dev/automaton $ ./bin/ctool

But that worked only after deleting install

rm -fr /usr/local/lib/python2.7/site-packages/automaton-1.0-py2.7.egg/

See also

~/dev/automaton $ python ./setup.py build install
env | grep $PATH
.bash_profile

testy, bash loop

for MODE in {"ECB","CFB","CBC","OFB"}; do ./dev/bin/test_roundtrip_encryption.sh $MODE 128 cassandra 2>/dev/null; done

9/20/16

emacs y-or-n-p and yes-or-no-p

TIL VLQ (variable-length quantity) encoding

9/22/16

~ $ brew outdated
emacs (24.5) < 25.1
openssl (1.0.2f, 1.0.2g, 1.0.2h, 1.0.2h_1) < 1.0.2i
homebrew/versions/openssl101 (1.0.1r, 1.0.1s, 1.0.1t_1) < 1.0.1u
~ $ brew upgrade openssl homebrew/versions/openssl101

resulted in

==> make install MANDIR=/Users/scottbale/.homebrew-clone/Cellar/openssl101/1.0.1u/share/man MANSUFFIX=ssl
==> Caveats
A CA file has been bootstrapped using certificates from the system
keychain. To add additional certificates, place .pem files in
  /usr/local/etc/openssl/certs

and run
  /usr/local/opt/openssl101/bin/c_rehash

This formula is keg-only, which means it was not symlinked into /usr/local.

Apple has deprecated use of OpenSSL in favor of its own TLS and crypto libraries

Generally there are no consequences of this for you. If you build your
own software and it requires this formula, you'll need to add to your
build variables:

    LDFLAGS:  -L/usr/local/opt/openssl101/lib
    CPPFLAGS: -I/usr/local/opt/openssl101/include
    PKG_CONFIG_PATH: /usr/local/opt/openssl101/lib/pkgconfig

9/23/16

List statistics of tip commit of all local branches (looking for a certain modified file)

git branch | xargs -I foo git lg --stat foo^!

9/25/16

Emacs 25.1

brew upgrade emacs

==> Caveats
Please try the Cask for a better-supported Cocoa version:
  brew cask install emacs

To have launchd start emacs now and restart at login:
  brew services start emacs
Or, if you don't want/need a background service you can just run:
  emacs

.app bundles were installed.
Run `brew linkapps emacs` to symlink these to /Applications.

installed to ~/.homebrew-clone/Cellar/emacs/25.1 (including application, drag to dock)

9/28/16

LDAP glossary:

  • CN = Common Name
  • OU = Organizational Unit
  • DC = Domain Component

10/1/16

pandoc pdflatex texlive on Mac OS X (why didn't I document this the first time?!)

first this:

pandoc: pdflatex not found. pdflatex is needed for pdf output.

googled, found: http://stackoverflow.com/a/22487657 http://tex.stackexchange.com/a/164047

then

[sbale ~ (mac-os-x)]$brew search texlive
Installing TeX from source is weird and gross, requires a lot of patches,
and only builds 32-bit (and thus can't use Homebrew dependencies)

We recommend using a MacTeX distribution: https://www.tug.org/mactex/

You can install it with Homebrew Cask:
  brew cask install mactex

first, clean up

/usr/local/texlive/2013basic/
/Library/TeX/Distributions/.FactoryDefaults/TeXLive-2013-Basic/
/usr/texbin symlink

now this:

pandoc: /Users/sbale/.cabal/share/pandoc-1.11.1/data/templates/template.latex: openBinaryFile:
does not exist (No such file or directory)

duh...have to run KungFuPandoc.sh from its directory

10/6/16

brew switch python 2.7.12

10/7/16

some problem w/ SASS task. (sass installed as part of rubygems, which in turn is installed as part of ruby, by homebrew)

~/dev/ripcord/opscenterd $ ./node_modules/.bin/gulp sass

where are npm global modules installed?

npm root
npm list -g

to get BSD ls (OSX is Berkley)

brew install coreutils

list w/o timestamps

gls -lh --time-style=+ dev/

homebrew must've screwed up both python and ruby?

brew switch ruby 2.3.1

10/11/16

our merge script failed until I did this

brew switch readline 6.3.8

curl admin session

curl -sSX POST -d '{"username":"foo","password":"******"}' 'http://localhost:8888/login'

returns

{"sessionid": "868e8c86ef0baeceb753f03e66c29af4"}

then

curl -sSH 'opscenter-session:868e8c86ef0baeceb753f03e66c29af4' 'http://localhost:8888/users/scottbale' | python -m json.tool
curl -sSH 'opscenter-session:868e8c86ef0baeceb753f03e66c29af4' 'http://localhost:8888/permissions/roles' | python -m json.tool
curl -sSX POST -H 'opscenter-session:868e8c86ef0baeceb753f03e66c29af4' -d '{"password":"test","role":"marketing"}' 'http://localhost:8888/users/George' | python -m json.tool

remove passwd.db periodically, it is git ignored

10/13/16

Craig found

./bin/opscenter -f | tee log/opscenter.log

10/14/16

auto generate oxygen dita docs

./bin/autogen-docs.sh

Build finished. The HTML pages are in _build/html-sphinxdoc.

10/18/16

brew|pip|python all messed up

$ brew config

See Homebrew/brew#1262 - moved Cellar from /usr/local/homebrew/Cellar to /usr/local/Cellar

$ brew info python
...
==> Caveats
Pip and setuptools have been installed. To update them
  pip install --upgrade pip setuptools


pip install --upgrade pip
sudo pip install --upgrade setuptools

Not sure this symlink is needed anymore:

$ ll /usr/local/homebrew
lrwxr-xr-x  1 scottbale  staff    32B Feb  1  2016 /usr/local/homebrew -> /Users/scottbale/.homebrew-clone

don't even get me started on ansible playbook...

SQLite Database Browser

10/19/16

JIRA search for "not closed" tickets:

  • use “resolution=unresolved"
  • then filter more by “status not in (foo, bar, yadayadayada)"

10/21/16

straightening out ccm. Looks like I pip-installed pcmanus one, and git-clone'd internal riptano one. Going to try uninstalling the first. update 11/1 looks like ccm is a dev dependency, see dev/requirements.txt, and it picks up the pcmanus one.

remember:

ctool list_repos

10/24/16

some manual pip installs needed for using openstack with automaton

sudo chown -R scottbale /usr/local/bin/
pip install python-novaclient python-keystoneclient funcsigs wrapt netifaces==0.10.5 positional monotonic

10/25/16

TIL tmux Meta, Ctrl-o swaps panes

TIL uptime

emacs see enabled color themes

C-h v custom-enabled-themes

10/26/16

:office: :parrot_conga: :parrot_conga: :parrot_conga: :parrot_conga: :parrot_conga: :parrot_conga: :parrot_conga: :parrot_conga: :parrot_conga:

bash trick

$ foo="roguebarbase"
$ echo $foo
roguebarbase
$ echo ${foo/bar/1}
rogue1base

10/27/16

git, change committer of HEAD

GIT_COMMITTER_NAME='Scott Bale' GIT_COMMITTER_EMAIL='[email protected]' git commit --amend
git log --format=fuller HEAD^!

10/28/16

github diff branch (example)

https://github.com/riptano/ripcord/compare/6.0.x...6946-ldap

10/31/16

DataStax academy DS201

Reading SSTable:

  • partition index (token, byte offset) (HDD)
  • summary index (RAM) - index into partition index
  • key cache (RAM) - bypasses indexes
  • bloom filter - partition is not, or might be, in this sstable (false positives are rare and tunable)

partition

  • grouping of data (single row or mult. rows)
  • ordering scheme for on disk
  • mergesort in memory
  • partition key yields token (from partitioner) - partition key is 1st value in primary key
    • additional piece(s) of PK is clustering column e.g. "Primary Key ((state), city, name, id)"

compaction

  • combines/merges sstables
  • drops tombstones after expired gc grace
  • drops drop records
  • anticompaction - incremental repair process of segregating repaired and unrepaired ranges into separate sstables

GC grace/tombstones? - About deletes:

  • deletion occurs during compaction, not immediately
  • C* column can have optional TTL expiration date. After expiry, column marked with tombstone. Tombstone exists for gc_grace_seconds (defaults to 10 days)
  • a deleted column can reappear due to entropy. Specifically, if a replica was down longer than gc grace period and did not receive delete.

vnodes

  • allows single physical node to have more than one distinct token range
  • 256 by default (pre C* 3.0)
  • post 3.0 much smaller default, configurable in .yaml conf file

replication

  • per keyspace, DC -

consistency - ONE, ALL, QUORUM, LOCAL_QUORUM

write path

  • 1st, to commit log (append-only on disk)
  • next, to MemTable (RAM, sorted according to data model)
  • periodically, MemTable flushed to disk SSTable (immutable)
    • corresponding commit log removed
  • eventually, you have multiple sstables on disk for a partition
    • last write wins - resolution protocol
    • needs compaction

new DSE 5.0.3 cluster:

$ ccm create dse-503-1 --dse --dse-username=your_username --dse-password=your_password -v 5.0.3 -n 3
Downloading http://downloads.datastax.com/enterprise/dse-5.0.3-bin.tar.gz to /var/folders/0l/mc......tar.gz (627.182MB)

11/1/16

containers: docker, lxc, lxd

11/2/16

to just run .clj tests on file save, similar to gulp watch:

lein quickie

11/7/16

FSA (doesn't roll over) vs HSA

11/9/16

homebrew has both n and nvm, competing node.js version managers (as opposed to just brew install node)

11/14/16

find empty directories [named foo]

find . -type d -empty [-name foo]

11/21/16

learned how, in emacs, to place Unicode combining characters (such as accents) on top of a character.

11/22/16

emacs regex alternative (have to escape the parens and pipe character)

\(remaining\|running\|completed\)_tasks

will match remaining_tasks, running_tasks, completed_tasks

11/25/16

Switching my phone over to cyanogenmod

  • 1Password Families requires Android 5.0 (Lollipop) or later
  • Samsung Galaxy S4 Mini, currently Android 4.4.x (KitKat)
  • Available: CyanogenMod (Android): 10.1 (4.2), 10.2 (4.3), 11 (4.4 KitKat), 12 (5.0 Lollipop), 12.1 (5.1), 13 (6.0 Marshmallow)

Checklist

  • disable 2FA (Google Chrome Authenticator app) (or use backup codes)
    • google accounts, dropbox: temporarily switched to Authenticator on old phone
    • AWS - had to deactivate, reactivate (prompted for two codes each time)
    • looks like Facebook no longer supports Google Authenticator - lame
    • github let me switch phones, but generated different backup codes :(
    • dnsimple - deactivate/reactivate, generated single recovery code, made me confirm it!
    • slack - looks like I only had 2FA enabled for teradata (not available for DS, disabled for Clojurians TODO)
  • unregister phone from everywhere (amazon, google, 1password, etc.)
  • backup contact info, photos
  • what's verizon voicemail #?
  • https://wiki.cyanogenmod.org/w/Install_CM_for_serranoltexx
  • reinstall apps
  • re-enable 2FA, 1password, etc.

11/28/16

I've given up on trying to install cyanogenmod on my Galaxy S4 Mini. Can't get heimdall to talk to the phone. Grrr.

  • put all 2FA back on to S4 Mini (sigh). Facebook does allow Google authenticator, just have to activate it by 16-digit key rather than scan barcode.
  • Next step: shop for a Google Nexus 5, 6, 5X or 6P (new or refurbished)

Unlocked Android phone on Verizon for Dummies

Phone RAM (GB) Ps (GHz) Sto (GB) Android Version Display (mm, res)
S4 Mini 1.5 1.7 8 4.4.2 KitKat 108, 960x540
Nexus 6 3 2.7 32, 64 5.0 Lollipop 151, 1440x2560
Pixel 4 2.15 32, 128 7.1 Nougat 130,
Galaxy S6 3 2.1 32, 128 5.0 Lollipop 130, 2560x1440

11/29/16

TIL EWMA (exponentially weighted moving average) chart

11/30/16

TIL FPGA; Amazon Lightsail VPS ("DigitalOcean killer")

12/2/16

DAC notes

  • CD (redbook) digital audio: 16-bit resolution, 44.1 kHz samling
  • higher quality digital audio commonly: 24-bit, 96 kHZ
  • FLAC - VLC player
  • bandcamp FLAC, exact quality is dependent on artist uploads - https://bandcamp.com/help/downloading
  • mp3 VBR (variable-bit encoding)
  • S/PDIF sounds pretty cool
  • Firewire IEEE 1394

12/9/16

Advice on long-term, robust, redundant storage of personal digital treasures (family pix, videos):

  1. Crashplan a. $9/mo a. memory intensive (stores hash of every block in RAM) a. initial backup slow, on order of weeks

  2. local a. ZFS raid-1 a. ZFS filesystem has nice properties built-in a. desktop (8 GB Ram, 4 would be fine), drives inside case? Nope: "I got a cardbus or whatever eSATA card and the drives are in a dual-bay eSATA case...(the eSATA raid-1 can push ~100MB/sec linear reads, but usb2 tops out at maybe 35MB/s)" a. recommend recent version of Ubuntu as host, for ease of integration with ZFS a. SAMBA for access by other OS (SAMBA provides cifs/smb) a. plex (browser-only interface for photos, videos, music)

12/14/16

In emacs dired mode, renamed a bunch of Clojure test files from test_foo.clj to foo_test.clj:

  • Pull up recursive dired of test src tree as per usual C-u, M-x, dired
  • Enter editable C-x, C-q
  • Regexp search/replace C-M-% with \(test\)_\([^\.]*\) and \2_\1
  • save C-x, C-s

Debug clj classpath:

(clojure.pprint/pprint (seq (.getURLs (java.lang.ClassLoader/getSystemClassLoader))))

start repl with test profile (to include in classpath :resource-paths defined for test profile)

lein with-profile +test repl :headless :port 2112

virtualenv is a new recent dev/requirements.txt addition (see dev/ansible/opscenterd_dev/tasks/main.yml)

pip install virtualenv

installed v 15.1.0

1/23/17

actually used virtualenv today finally (for a sandboxed C* driver example program)

~/dev/py $ virtualenv venv
~/dev/py $ source venv/bin/activate
(venv) ~/dev/py $ pip install cassandra-driver
(venv) ~/dev/py $ deactivate

to remove, just delete folder

~/dev/py $ rm -fr venv

cql remove all rows from a table

TRUNCATE table;

fix for Too many open files in system (specific to OS X Yosemite)

First, increase the kernel limits:

echo kern.maxfiles=65536 | sudo tee -a /etc/sysctl.conf
echo kern.maxfilesperproc=65536 | sudo tee -a /etc/sysctl.conf

Next, created this file to increase launchctl limit of max open files

sudo cat /Library/LaunchDaemons/limit.maxfiles.plist

Then rebooted, and check various things (apparently, ulimit is subservient to launchctl):

sysctl -A | grep kern.maxfiles
launchctl limit maxfiles
ulimit -a

2/20/17

lein installed by ansible playbook - just downloads stable version

  • upgraded from 2.5.3 to 2.7.1 - fixed my emacs cider|nrepl|cider-nrepl 0.2.10|0.2.12 incompatibility

3/31/17

-922337203685478 (aka -2^63) is the min token when using Murmur3 partitioner

4/20/17

thrift nonsense

pip show thrift
pip uninstall thrift
brew remove thrift
brew install thrift090 --with-python --with-java
brew options thrift090
# echo 'export PATH="/usr/local/opt/[email protected]/bin:$PATH"' >> ~/.bash_profile
brew link --force thrift090

6/21/17

the highest thread(s) consuming the highest CPU

top -H -p <pid>

6/22/17

check out trapper keeper, built on top of Clojure component

6/29/17

Create ~/.lein/credentials.clj.gpg to replace profiles.clj (use passphrase when creating gpg key)

brew install gpg
echo '{#"https://repo.datastax.com/" {:username "user" :password "encrypted-password-from-artifactory"}}' > ~/.lein/credentials.clj
gpg --generate-key
gpg --default-recipient-self -e ~/.lein/credentials.clj > ~/.lein/credentials.clj.gpg

troubleshooting

gpg -k
gpg --quiet --batch --decrypt ~/.lein/credentials.clj.gpg
lein help gpg

7/26/17

mocking time.time() in python unit test

In file being tested:

In test file (patch has to go before defer)

import time

@patch('time.time', Mock(return_value=started_ts))
@defer.inlineCallbacks
def test_launch_codez(self):

8/21/17

had to upgrade node to 6 (but not to 8!)

npm -version
brew upgrade node 
brew outdated
node --version
npm info node
brew install node@6
node --version
npm info node
brew switch node 6.9.1
node --version

8/31/17

TIL top for single process

top -p PID

or on MAC

top -pid PID

9/6/17

Unlocked Android phone on Verizon for Dummies, revisited (see 11/28/16)

Android OTA (over-the-air) updates is of interest. Google-branced devices get stock Android OTA updates direct from Google. Others, e.g. Motorola, get OTA updates from the respective hardware vendor? https://source.android.com/devices/tech/ota/

| | | | | | Will be OTA upgraded | |

Phone RAM (GB) Ps (GHz) Sto (GB) Android Version to Android Version Display (mm, res)
S4 Mini 1.5 1.7 8 4.4.2 KitKat 108, 960x540
Nexus 6 3 2.7 32, 64 5.0 Lollipop 7.0 Nougat 151, 1440x2560
Nexus 5x
Nexus 6P
Pixel 4 2.15 32, 128 7.1 Nougat 130,
Galaxy S6 3 2.1 32, 128 5.0 Lollipop 130, 2560x1440
Moto E4 Plus 2 4.13 16 7.1.1 Nougat 130, 1280x720

https://en.wikipedia.org/wiki/Comparison_of_Google_Nexus_smartphones

Looks like my Samsung S4 Mini has a micro SIM card, and Nexus 5x needs a nano SIM

9/9/17

2FA reset Google authenticator - I need to reset wife's old iphone and then re-init Google authenticator. Some accounts (like google) I'm going to use existing backup codes. Others I'm going to temporarily disable.

  • 4 google accounts - backup codes
  • Amazon AWS - deactivate
  • Dropbox - backup code
  • GitHub - backup code
  • Facebook - disable
  • DNSimple - disable
  • Okta - disable
  • Datastax Academy Slack - backup codes

9/13/17

can't remember in iTerm prefs why I mapped right command key to option. Going to revert back and see if I remember...

Now I remember (2/3/16):

  • opt-f, opf-b forward backward navigation in terminal, e.g. editing command line.
  • tmux opt-v to page back through tmux history
  • navigation in zile

use eslint, jscs in emacs

  • npm install -g eslint jscs jscs-jsdoc
  • install flycheck package
  • install exec-path-from-shell package (related: C-h v exec-path)
  • stuff in init.el

9/17/17

gulp watch

~/dev/ripcord/opscenterd $ ./node_modules/gulp/bin/gulp.js watch

9/18/17

note to self: git nuke nuked my .dir-locals.el

9/19/17

TIL from Tom U

"$(npm bin)"/gulp …
jscs --config src/js/etc/jscsrc src/js/ripcord/widgets/foo/bar.js

❯ whence -c npm-exec
npm-exec () {
        if ! hash npm 2> /dev/null
        then
                return
        fi
        local bin="$1"
        shift
        "$(npm bin)"/"$bin" "$@"
}

9/20/17

iterm keybindings revisited: (See 2/3/16, 9/13/17)

My left thumb is hurting from too much cmd-tab switching between apps in OS X. So I want to use my right command key for that. Problem is, in iterm (in bash shell) I also want to keep using the alt- commands I'm accustumed to.

Solution:

  • in iTerm preferences
    • Profiles->Keys: Option should send +Esc (as before)
    • Keys: Change right command key back to command
    • Keys: +, then add cmd-b and "send escape sequence" and set to Esc plus "b".
    • Do that to mimic other desired alt commands

9/26/17

TIL

  • commons-logging - an API
  • log4j - implementation, common provider of commons-logging
  • slf4j - an API, successor to commons-logging
  • logback - implementation, common provider of slf4j
  • bridges: jar files containing commons-logging classes reimplemented to delegate to slf4j

9/28/17

sudo pip install --upgrade setuptools==33.1.1

9/29/17

Great subtle python bug:

@defer.inlineCallbacks
def _gatherDeadline(self, ks, table):
    deadline = yield self.data_fetcher.getDeadlineSeconds(ks, table),
    defer.returnValue([{
        "name": "deadline",
        "value": deadline,
        "units": "seconds"
    }])

trailing comma in 1st line of method makes deadline a tuple, not a Deferred! yielding on a tuple just returns the tuple, apparently!

10/3/17

remotes have conflicting branch names, so create a new branch with diff name that tracks the upstream branch

~/dev/java-driver-dse $ git branch --track 1.x.upstream upstream/1.x

10/5/17

flame graph

10/11/17

tupton:

if your ccm stop is having trouble killing your multi node cluster, pkill -9 -f cassandra is pretty useful better than the ol’ ps aux | grep cassandra, copy and paste, and kill -9 <pids> business that I used to do pgrep is also nifty, instead of ps … | grep … pgrep -f to search the full command line including arguments

10/16/17

fast 'n loose grep for IP addresses in a file

grep -E "\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b" temp.json | less

10/18/17

TIL from Tom U: Quicktime, screen recording, export, ffmpeg, gifsicle

ffmpeg -i ~/Desktop/metrics-dialog.mov -pix_fmt rgb24 -r 10 -f gif - | gifsicle --optimize=3 --delay=7 > ~/Desktop/metrics-dialog.gif

11/2/17

TIL about natural sort ordering

asynchronous IO reactor (Java driver uses Netty) o_nonblock

11/27/17

reading about

  • rg instead of ack or ag
  • vim fzf uses rg
  • ripgrep and vim-grepper

2/21/18

upgrade cassandra python driver in virtual environment

https://github.com/datastax/python-driver

(venv) ~/dev/py $ pip list
(venv) ~/dev/py $ pip help install
(venv) ~/dev/py $ pip install -U cassandra-driver
(venv) ~/dev/py $ python dataflate.py

3/1/18

groovy-mode for .gradle file syntax emacs highlighting (not gradle-mode)

migrating from lein repl to gradle-clojure plugin

https://github.com/gradle-clojure/

~/dev/ripcord $ ./gradlew :agent:nrepl

with gradlew :agent:nrepl

;; Connected to nREPL server - nrepl://localhost:7888
;; CIDER 0.16.0 (Riga), nREPL 0.2.8
;; Clojure 1.8.0, Java 1.8.0_65
;;     Docs: (doc function-name)
;;           (find-doc part-of-name)
;;   Source: (source function-name)
;;  Javadoc: (javadoc java-object-or-class)
;;     Exit: <C-c C-q>
;;  Results: Stored in vars *1, *2, *3, an exception in *e;
WARNING: CIDER requires nREPL 0.2.12 (or newer) to work properly
More information.
WARNING: CIDER's version (0.16.0) does not match cider-nrepl's version (nil). Things will break!
More information.
user>

with lein repel

;; Connected to nREPL server - nrepl://localhost:2112
;; CIDER 0.16.0 (Riga), nREPL 0.2.12
;; Clojure 1.8.0, Java 1.8.0_65
;;     Docs: (doc function-name)
;;           (find-doc part-of-name)
;;   Source: (source function-name)
;;  Javadoc: (javadoc java-object-or-class)
;;     Exit: <C-c C-q>
;;  Results: Stored in vars *1, *2, *3, an exception in *e;
user>

Tom U says fzf is greatest dev tool ever

❯ cat ~/.local/bin/git-fuzzy-branch
#!/usr/bin/env zsh

git branch | sed "s/^[\* ] //" | fzf --extended --multi --exit-0 --select-1 --query "$@" --preview="git log {}"

3/7/18

Investigating Gradle clojure cider-nrepl support

  • tools.nrepl
    • server.clj defines defn start-server, defn default-handler
  • leiningen
    • repl.clj defn- server-forms builds handler, overrideable by plugins (e.g. for additional middleware)
    • defn- handler-for uses nrepl-middleware from project.clj
  • cider-nrepl (additional middleware for tools.nrepl)
    • plugin.clj (lein plugin) adds its own middleware to :repl-options :nrepl-middleware of project.clj
  • gradle-clojure (simple gradle clojure plugin)
    • clojure_nrepl.clj - simply calls nrepl/start-server only specifying port - no hook to override middleware

8/29/18

regex non characters

  • \d - digits
  • \w - alphanumeric
  • \W - non-alphanumeric
  • \s - whitespace
  • \S - non-whitespace

10/12/18

APUE - Bulat's review recommend 1-5 - foundational 8-11 super important from POV of software development (not necessarily OS, systems) - help w/ perspective on Java stuff

  1. unix overview
  2. file I/O (atomic operations)
  3. process env
  5. process relationships - groups, leaders, signal propogation, job control
  6. signals - super important, especially writing servers, signal handling in "reentrant" way
  7. Threads - signal handlers in threads
  8. nonblocking IO (builds on 5. IO)

recommended specs: HTTP (1.1), TCP, DNS, EcmaScript

Fighting complexity: Systems Theory

  • Stephen Wolfram: A New Kind of Science

blocking DNS - InetAddress.getHostByName

blog - Mechanical Sympathy (building software that is CPU, disk, memory aware)

3/29/19

installed Java 12 on Mac - Oracle JDK

  • download and install DMG file (installs to /Library/Java/JavaVirtualMachines/jdk-12.jdk/)
  • update ~/.bash_profile which exports JAVA_HOME

3/30/19

short-term TODO

  • Java
  • organize dev-log into multiple .md files by year?
  • dotfiles 2.0 organization
    • master should have light, linux-style dotfiles (as would work well on new aws instances)
    • branch for Mac OS X
    • branch for full emacs config for
  • rejigger emacs config after the style of Rob
  • install Ubuntu VirtualBox

Java

  • Oracle's Java SE 9 JDK has reached EOL, is no longer available for download
  • Oracle commercial subscription model and 6-month release cadence began with JDK 10
  • OpenJDK binaries still offered by Oracle under GPL (with linking exception?)
  • https://www.azul.com/eliminating-java-update-confusion/

my quest to locally build Gradle documentation

~/dev/gradle $ export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home
~/dev/gradle $ java -version
java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)
~/dev/gradle $ ./gradlew clean

...

* What went wrong:
Failed to apply plugin [id 'gradlebuild.available-java-installations']
> Could not create an instance of type org.gradle.gradlebuild.java.AvailableJavaInstallations_Decorated.
   > Building gradle on Java 7/8 requires java9Home system property or project property

lead me down a rabbit hole of trying to install a newer version of Java on this MacBook, then trying to understand Oracle's current Java licensing and release model, then beginning the investigation of installing VirtualBox since screw OS X for development... then I gave up, because I was only trying to quickly build a local copy of gradle documentation for the flight home, which was yesterday as I write this.

4/1/19

TIL option-click notifications icon to put OS X in do not disturb mode

New VirtualBox Ubuntu vm, OS X host

  • VirtualBox 6.0.4
  • Ubuntu 18.04.3 LTS Bionic Beaver (download was an .iso file)
  • create 80 GB fixed disk size; 4096 Megs RAM
  • sudo apt-get -y install gcc make perl
  • after creating and booting vm, Devices->Optical Drive->Insert Guest Additions CD Image, this auto-runs in the vm, builds guest addition kernel modules
  • install git via ppt
  • create new keypair
  • enable bidirectional shared clipboard (under Devices)
  • install java openjdk-8-jdk
  • install emacs25

Notes

4/11/19

brew cask list emacs

TIL jna, jnr, jnr-posix, sigar, oshi

more ubuntu vm setup

  • emacs config
  • cloned work repo
  • dotfiles - didn't mess with stock .bashrc or .profile - stock .profile sources .bashrc, which sources my .bash_aliases dotfile

4/25/19

keybindings (See 2/3/16, 9/13/17, 9/20/17)

software/programs whose keybindings I care about

  • emacs (Ctrl where caps lock is, meta keys on either side of space bar)
  • tmux (set prefix to C-])
  • bash command line - subset of emacs navigation plus basic editing, undo
    • C-a M-b C-b C-l C-f M-f C-e C-d M-d C-k C-y C-/ C-p C-n
  • xmonad (custom bindings)
    • left window key as meta key
  • all OS
    • Make caps lock another Ctrl
    • key-to-left-of-spacebar+spacebar - make spotlight (or equivalent) appear
  • OS X (preferences->keyboard->Modifier Keys)
    • ⌘-w, ⌘-q, ⌘-c, ⌘-v, ⌘-x, ⌘-tab, ⌘-,
    • swap ctrl, opt keys (gives me a right ctrl key) on MBP keyboard
    • magic opt-⌘-
    • mash ctrl-opt-⌘-
  • linux
    • C-w, C-q, C-c, C-v, C-x, alt-tab
    • by default, opt seems to act as stock M- key
  • Virtualbox
    • left ⌘ is escape key
    • swapped ctrl & opt keys in host OS X is reflected in guest Ubuntu
    • in guest Ubuntu, ⌘ by default seems to operate as s- aka "super" (at least in Emacs), equivalent to "windows" key

TIL super key https://en.wikipedia.org/wiki/Super_key_(keyboard_button)

Virtualbox View->Virtual Screen 1 when docked seems to prefer zoom 200%. On MBP I noticed it wouldn't display if I sized the window bigger.

I forgot that, in OS X preferences, I have the Apple internal keyboard and the Kinesys freestyle configured differently.

The bottom row of keys is what's critical

  • Apple internal
    • bottom row: fn, ctrl, alt|option, ⌘, space, ⌘, alt|option
    • Caps lock -> ^ ctrl
    • Ctrl ^ -> option
    • option -> ^ ctrl
    • cmd ⌘ -> ⌘ cmd
    • bottom row is effectively: opt, ctrl, ⌘, space, ⌘, ctrl
    • for (guest) ubuntu it is: alt, ctrl, super, space, super, ctrl
  • Kinesys Freestyle
    • bottom row: ctrl, window, alt, space, alt, ctrl
    • Caps lock -> ^ ctrl
    • Ctrl ^ -> ^ ctrl
    • option(alt) -> ⌘ cmd
    • cmd ⌘ -> option(alt)
    • os x recognizes window key as ⌘ cmd (see prev wikipedia link); alt key as option
    • bottom row is effectively: ctrl, opt, ⌘, space, ⌘, ctrl
    • for (guest) ubuntu it is: ctrl, alt, super, space, super, ctrl

Further complicating things is OS X emacs config

;; OS X meta keys
(setq mac-option-key-is-meta nil)
(setq mac-command-key-is-meta t)
(setq mac-command-modifier 'meta)
(setq mac-option-modifier nil)

What I want for (guest) Ubuntu is

  • bottom row: ctrl, window (super), alt, space, alt, ctrl - which means I may have to make changes to (host) OS X keybindings once I'm ready to switch to VM?
  • kinesys should be unmodified
  • Apple internal should be
    • Caps lock -> ^ ctrl
    • Ctrl ^ -> ^ ctrl
    • option -> ⌘ cmd
    • cmd ⌘ -> option
    • making bottom row effectively: ctrl, ⌘ (super), alt, space, alt, ⌘ (super) - so I lose right ctrl but that's tolerable

install gnome-do (quicksilver/spotlight equivalent)

4/26/19

Dojo modules

  • dojo - dojo 1
  • dijit - dojo 1 UI library
  • dojox - "extras library", contains DataGrid
  • dgrid - SitePen?
  • dojo-util - build utilities

4/29/19

sudo apt-get virtualbox-guest-utils virtualbox-guest-dkms virtualbox-guest-x11

5/3/19

moving to virtualbox

  • switch host modifier keys (see above)
  • install linux (beta) slack app
  • mount shared host folder ~/code https://gist.github.com/estorgio/1d679f962e8209f8a9232f7593683265
  • did more research on guest display issues on Macbook native display. What I'm reading is it's due to retina HiDSP display, and I just need to manually switch it to scaled 200%

Summarizing modifier key settings for each of the two keyboards, for either using native OS X or ubuntu VM (since I'll have to go back and forth at times)

  • native OS X
    • Apple internal keyboard (swap option and ctrl)
      • Caps lock -> ^ ctrl
      • Ctrl ^ -> option
      • option -> ^ ctrl
      • cmd ⌘ -> ⌘ cmd
    • Kinesys Freestyle (swap option and cmd)
      • Caps lock -> ^ ctrl
      • Ctrl ^ -> ^ ctrl
      • option(alt) -> ⌘ cmd
      • cmd ⌘ -> option(alt)
  • Ubuntu VM
    • Apple internal keyboard (swap option and cmd)
      • Caps lock -> ^ ctrl
      • Ctrl ^ -> ^ ctrl
      • option -> ⌘ cmd
      • cmd ⌘ -> option
    • Kinesys Freestyle (unmodified except for Caps lock)
      • Caps lock -> ^ ctrl
      • Ctrl ^ -> ^ ctrl
      • option(alt) -> option(alt)
      • cmd ⌘ -> ⌘ cmd

5/6/19

@#^!$!@!! The "mount shared host folder" steps I followed above caused my vm to boot into emergency mode! I had to, from there, edit /etc/fstab and remove the line I added. When I rebooted it came up fine. Don't believe everything you read on the internet!

export git stash as patch

git stash show -p > ../git-stash-patch

import

git apply git-stash-patch
git stash

add self to vboxsf group

sudo usermod -aG vboxsf scott

or

sudo adduser scott vboxsf

vbox shared folder auto-mounted to

/media/sf_<name_of_folder>

manually mount (not sure about the uid and gid flags)

sudo mount -t vboxsf -o uid=1000,gid=1000 sharename /home/<user>/vboxshare

increased CPU count from 1 to 4 for ubuntu vm

5/16/19

show emacs line numbers in margin: M-x linum-mode (Not to be confused with line-number-mode and column-number-mode)

7/12/19

JavaScript asynchronous module loading, import vs require

9/6/19

NVMe - Non Volatile Memory Express - host controller interface & protocol designed for interfacing SSD over PCIe bus (faster than SATA)

10/3/19

Ubuntu Full Disk Encryption (FDE)

  • eCryptfs - claimed to be buggy now
  • fscrypt
  • LUKS (Linux Unified Key Setup) - only if erase hard disk on Ubuntu installation
  • LVM (logical volume management)

new Ubuntu 19.04 install

prepare live SD card

  • download .iso, verify checksum

      echo "2da6f8b5c65b71b040c5c510311eae1798545b8ba801c9b63e9e3fd3c0457cbe *ubuntu-19.04-desktop-amd64.iso" | shasum -a 256 --check

  • format SD card (from OS X, to be booted by PC) - results in (expected) error message "disk not readable"; eject SD

      hdiutil convert ubuntu-19.04-desktop-amd64.iso -format UDRW -o ubuntu-19.04-desktop-amd64.img
  diskutil list
  diskutil unmountDisk /dev/disk2
  sudo dd if=ubuntu-19.04-desktop-amd64.img.dmg of=/dev/disk2 bs=1m

installation

  • Dell Precision boot menu - F12

      Boot mode is set to: UEFI; Secure Boot: ON

  • UEFI (Unified Extensible Firmware Interface)

  • System settings - F2

  • Had to enable secure boot from card reader

    • F2 - system settings
    • in the BIOS, go to System Configuration, Miscellaneous Devices
    • Enable Secure Digital (SD) Card Boot
    • apply, exit
    • F12 into one time boot menu

Glossary

  • UEFI (Unified Extensible Firmware Interface)
  • GPT (GUID Partition Table) - disk partitioning scheme, alternative to PC's MBR (Master Boot Record)
  • GRUB (GNU GRand Unified Bootloader) - 2nd stage boot loader
  • NVMe disk

10/7/19

Dell Precision 5530, x64-based

  • intel i7 - 6 core, 2.60GHz
  • 32.0 GB RAM
  • secure boot on
  • BIOS mode - UEFI

GPT partition

  • /boot partition unencrypted

TIL

  • For disk name sda, sd stands for "SCSI device" for historical reasons. "a" is assigned to the first disk. 2nd disk often named sdb, etc. Child partitions are named sda1, etc.

10/8/19

linux stuff

  • swapon -s to get summary of swap info
  • lsblk list block devices

10/14/19

TIL ubuntu

  • super hotkey is equivalent to ubuntu "Activities" (upper-left corner)
  • super-a hotkey launches "Show Applications" (bottom-left corner) which is sort of like Spotlight or gnome-do
  • super takes you back to starting point

10/15/19

FHS - Filesystem Hierarchy Standard (standard linux directory structure)

10/17/19

TIL ubuntu

  • super+up maximizes window
  • super+down restores window
  • super+left|right splits view to left|right

So I have complete spotlight|gnome-do like behavior just in stock ubuntu, using the super key. (Unfortunately, super is my xmonad meta key, so if I ever want to use xmonad again I'll have to figure out a new meta key.)

Downloaded clojure boot

scott@scott-Precision-5530:~/dev/ripcord$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
scott@scott-Precision-5530:~/dev/ripcord$ sudo bash -c "cd /usr/local/bin && curl -fsSLo boot https://github.com/boot-clj/boot-bin/releases/download/latest/boot.sh && chmod 755 boot"
[sudo] password for scott: 
scott@scott-Precision-5530:~/dev/ripcord$ boot
Downloading https://github.com/boot-clj/boot/releases/download/2.7.2/boot.jar...
Running for the first time, BOOT_VERSION not set: updating to latest.
Retrieving clojure-1.8.0.pom from https://repo1.maven.org/maven2/ (8k)
Retrieving oss-parent-7.pom from https://repo1.maven.org/maven2/ (5k)
Retrieving maven-metadata.xml from https://repo.clojars.org/
Retrieving boot-2.8.3.pom from https://repo.clojars.org/ (2k)
Retrieving boot-2.8.3.jar from https://repo.clojars.org/ (2k)
Retrieving clojure-1.8.0.jar from https://repo1.maven.org/maven2/ (3538k)
#http://boot-clj.com
#Thu Oct 17 09:12:13 CDT 2019
BOOT_CLOJURE_NAME=org.clojure/clojure
BOOT_VERSION=2.8.3
BOOT_CLOJURE_VERSION=1.8.0

10/23/19

I realized today I'm used to the OS X scrolling, so in ubuntu settings I disabled "natural scrolling". Feel dirty, need a shower.

10/24/19

fix the emacs gnu package signature stuff

$ gpg --homedir ~/.emacs.d/elpa/gnupg  --keyserver keyserver.ubuntu.com --recv-keys 066DAFCB81E42C40

http://elpa.gnu.org/packages/gnu-elpa-keyring-update.html https://www.reddit.com/r/emacs/comments/d9rchm/emacs_archivecontentssig_not_verifying/

11/7/19

Can't get old ViewSonic monitor with DVI output to work on ubuntu 19.04 with DVI-to-HDMI cable https://superuser.com/a/1271463

11/15/19

TIL about /sys/devices/system/cpu/vulnerabilities/ which contains a list of known CPU vulnerabilities and their mitigation statuses

5/21/20

ImageMagick

apt-get install imagemagick
convert -resize 50% -delay 100 -loop 0 *.jpg sm.gif

11/18/20

From Clojurians #clojure

  • brew tap for clojure/tools/clojure maintained by the Clojure team

  • homebrew core tap is now being kept up to date with stable releases from clojure/tools/clojure

  • linuxbrew tap tracks homebrew core tap, ~ couple day lag

  • debian clojure package just wraps clojure.jar, and has none of the cli features

  • https://clojure.org/guides/getting_started#_clojure_installer_and_cli_tools

  • clojure, clj

      clojure -Sdescribe

both clojure, lein use .m2 to cache deps

~/.clojure/deps.edn https://clojure.org/reference/deps_and_cli#_deps_edn_sources

11/22/20

TIL in emacs, open a log file in a buffer then M-x auto-revert-mode set to true, to effectively tail the log. https://stackoverflow.com/a/19589885/2495576

11/23/20

TIL in tmux Meta-s brings up sort window (accidentally hit that while trying to chord Meta : then swap-window)

11/30/20

emacs TIL while debugging

Error retrieving: https://elpa.gnu.org/packages/archive-contents (error http 400)

while attempting package refresh

Examine (C-h v) variables:

  • package-archives - set|modified by my setup-packages.el

  • package-check-signature

  • system-configuration-features (should have GNUTLS in it)

    • M-: (string-match-p "GNUTLS" system-configuration-features)

  • M-x toggle-debug-on-error before M-x package-list-packages:

    Result from *Backtrace* buffer:

      url-http-generic-filter(#<process elpa.gnu.org> "HTTP/1.1 400 Bad Request\015
  Date: Mon, 30 Nov 2020 15:43:38 GMT\015
  Server: Apache/2.4.38 (Debian)\015
  Content-Length: 441\015
  Connection: close\015
  Content-Type: text/html; charset=iso-8859-1\015
  \015
  <!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">
  <html><head>
  <title>400 Bad Request</title>
  </head><body>
  <h1>Bad Request</h1>
  <p>Your browser sent a request that this server could not understand.<br />
  Reason: You're speaking plain HTTP to an SSL-enabled server port.<br />
   Instead use the HTTPS scheme to access this URL, please.<br />
  </p>
  <hr>
  <address>Apache/2.4.38 (Debian) Server at elpa.gnu.org Port 443</address>
  </body></html>
  ")

12/1/20

Trying to install scala-mode emacs package

  • first, complained about invalid PGP signature
    • toggled debug error on, see above
    • saw package.el was checking for a non-nil value of package-check-signature
    • changed package-check-signature value to nil (was 'allow-unsigned')
  • now, complaining about invalid TAR header, I give up

12/9/20

emacs stuff

Finally figured out my package install woes (attempting to install scala-mode): my melpa URL was out of date (in setup-package.el). https://melpa.org/#/getting-started has current URLs. Also removed the archive at ~/.emacs.d/elpa/archives/melpa/ and bounced emacs (not sure if that was necessary). Then either M-x package-refresh-contents or my usual M-x package-list-packages.

Also, I first (not sure if this is necessary) installed gnu-elpa-keyring-update see https://stackoverflow.com/a/58211420/2495576.

Upgrade a bunch of other packages including cider. As a reminder to self, then had to go update version of cider-nrepl to match, in both ~/.lein/profiles.clj and ~/.gradle/gradle.properties.

[later] reviewing all things nrepl:

  • notes 6/2/16 basic summary
  • notes 3/7/18 re: gradle clojure cider-nrepl support (became ticket OPSC-14321)
  • tools.nrepl successor is nrepl (0.3.1 should be drop-in replacement for 0.2.13)
  • reply aka REPL-y - a replacement for stock Clojure repl, unrelated to nrepl (but with integration)

12/12/20

https://clojurians.slack.com/archives/C03S1KBA2/p1607694968375800

https://clojure.org/reference/refs says Clojure refs implement snapshot isolation. Is somewhere stated what kind of concurrency issues this isolation level (specifically in Clojure code) can cause? Does anyone experienced such issues in practice?

The concurrency issue you are most likely to see with refs is write skew (because read-only refs are not part of the default ref set that can cause a transaction to retry). But that’s easily worked around when it’s an issue by using ensure instead of deref to add the ref to the ref set even on read.

12/22/20

Just accidentally learned, in markdown mode, you can place cursor on a header row, hit tab and collapse/expand the document tree. (Was trying to alt-tab between open windows I think.)

1/31/21

Concatenative languages

4/27/21

Emacs M-x package-list-packages, "Waiting for refresh to finish..." hanging

  • TIL M-x eww RET https://www.wikipedia.org/ to check whether TLS is working
  • gnutls-algorithm-priority variable, set to "normal:-vers-tls1.3"? Doesn't have any effect.
  • I give up

6/14/21

Got a message that cl package is deprecated as of emacs 27.1. Found cl-macs package. TIL on macOS, emacs 27.2, libraries (source and compiled) are located in

/Applications/Emacs.app/Contents/Resources/lisp/emacs-lisp/

e.g.

/Applications/Emacs.app/Contents/Resources/lisp/emacs-lisp/cl-macs.el.gz
/Applications/Emacs.app/Contents/Resources/lisp/obsolete/cl.el.gz

compiled byte code

/Applications/Emacs.app/Contents/Resources/lisp/obsolete/cl.elc

See usage of cl in my setup-package.el

1/12/23

1password issue - other users can't open the app (electron ffmpeg)

symptom is application fails to launch with a message

1Password cannot be opened because of a problem. and Ignore and Report buttons

report includes this clue:

Dyld Error Message:
  dyld: Using shared cache: 1FF61C30-FB8C-36BE-B7E2-5D513370F4BB
Library not loaded: @rpath/libffmpeg.dylib
  Referenced from: /Applications/1Password.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Electron Framework
  Reason: image not found

workaround:

cd /Applications/1Password.app/Contents/Frameworks/Electron\ Framework.framework
sudo chmod -h 755 ./Electron\ Framework ./Helpers ./Libraries ./Resources
cd Versions
sudo chmod -h 755 ./Current

I did the workaround without the sudo