Speed Issues

[ekinakyurek]

1. CuDNN-RNN sorting requirement:

To call CuDNN, we need to sort sequences by length. This breaks the order of sentence, so we need to reorder hidden states after the call. This might be a slow procedure, we might come up with an alternative. Any ideas @denizyuret ? I guess we can do padding from right to make sequences same length, and process them at once. And make the embeddings of pads 0, and expect RNN to learn to ignore those 0s.

Morse.jl/src/models.jl

Line 76 in 2dfe277

return out.hidden[:,:,end][:,indices], out.memory[:,:,end][:,indices]

2. Minibatching at decoding time:

1- We can't process all words at once because of the output encoding. Should we stick to non output-encoding version of the model for faster public model? @denizyuret

2- I believe we should be able to eliminate for loop over the characters of a word (for t=1:timeSteps). Is there any obstacle for that? I guess this is only helpful in training

Morse.jl/src/models.jl

Lines 284 to 306 in 2dfe277

	for w=1:wordNumber
	dh2 = (hiddens2[1][:,w:w] .+ hiddens3[1], hiddens2[2][:,w:w] .+ hiddens3[2])
	dh1 = (hiddens1[1][:,w:w], hiddens1[2][:,w:w])
	input = v.specialIndices.bow
	outEmbedding = [];
	continueStoring = true
	for t=1:timeSteps
	rnninput = M.outputEmbed([input])
	if continueStoring && input == v.specialIndices.eow
	continueStoring = false
	elseif continueStoring && input ∉ v.specialIndices && length(v.tags[input]) > 1
	push!(outEmbedding,rnninput)
	end
	dh1, dh2 = M.decoder(rnninput, dh1, dh2; training=false)
	h2 = dh2[1]
	scores = M.output(reshape(h2, size(h2,1), size(h2,2)))
	#prediction
	input = argmax(convert(Array,scores))[1]
	preds[w,t] = input
	# loss calculation
	total += M.loss(scores, d.seqOutputs[w:w,t] .* d.masks[w:w,t]; average=false)

[denizyuret]

On Tue, Dec 22, 2020 at 6:37 PM Ekin Akyürek ***@***.***> wrote: *1. CuDNN-RNN sorting requirement:* To call CuDNN, we need to sort sequences by length. This breaks the order of sentence, so we need to reorder hidden states after the call. This might be a slow procedure, we might come up with an alternative. Any ideas @denizyuret <https://github.com/denizyuret> ? https://github.com/ai-ku/Morse.jl/blob/2dfe277275ae2087887a8816ce2d3e7add04394a/src/models.jl#L76

Two alternatives: 1. Just use padding and do not sort. 2. I am working on a new cuDNN interface which includes a builtin "padded array" option: https://github.com/denizyuret/CUDA.jl/tree/dy/cudnn/lib/cudnn -- It is a PR under evaluation now, probably will be released with Julia 1.6.

*2. Minibatching at decoding time:* 1- We can't process all words at once because of the output encoding. Should we stick to non output-encoding version of the model for faster public model? @denizyuret <https://github.com/denizyuret>

This is during training, right? Transformers solve this using masking but I guess something similar in RNNs may not be possible. How much do we lose without output encoding? We can certainly make this an option.

2- I believe we should be able to eliminate for loop over the characters of a word (for t=1:timeSteps). Is there any obstacle for that? https://github.com/ai-ku/Morse.jl/blob/2dfe277275ae2087887a8816ce2d3e7add04394a/src/models.jl#L284-L306

I don't think so.

…

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#16>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAN43JSGRM6WLIIKSHC2UDDSWC4JXANCNFSM4VFZHZKA> .